2.1 The Reporting Function

Oswald (Reference Oswald2008) critiques the subjective well-being and happiness literature by arguing that this research has yet to establish the shape of the function relating reported subjective well-being to actual well-being. Oswald (Reference Oswald2008) states:

A similar argument is easily applied to other ordinal variables, such as happiness, satisfaction, trust, and measures of quality. Standardized test scores perhaps require a brief explanation. As discussed in Bond and Lang (Reference Bond and Lang2013), standardized test scores may not have a known or fixed interval between values.Footnote ⁷ Consider a simple case where a test score simply assigns values based on the number of questions answered correctly by each student. If some questions are more difficult than others, then assuming a fixed interval or cardinal scale may not be valid. Since test scores approximate student “learning,” answering difficult questions correctly may signal a larger marginal gain in learning than answering the easier questions correctly (see, e.g., Reardon Reference Reardon2008; Nielsen Reference Nielsen2017). The reporting function for test scores, therefore, defines the relationship between actual student learning to performance on a test.

2.2 Sufficient Conditions for Robustness of Sign

Testing for the robustness of monotonic increasing transformations is complicated by the fact that there are an infinite number of ways to transform an ordinal variable. Precisely, due to this reality, Schröder and Yitzhaki (Reference Schröder and Yitzhaki2017) derive two theoretical conditions under which effect estimates will be robust in sign to monotonic increasing transformations. The first condition refers to the robustness of mean comparisons between groups, and the second condition refers to the robustness of OLS regression estimates.Footnote ⁸

These conditions draw from the literature on stochastic dominance (Hadar and Russell Reference Hadar and Russell1969). In particular, the first condition states that the mean of one group is larger than that of another mutually exclusive group if and only if the former first-order stochastically dominates the latter. This condition implies that if the cumulative distribution functions (CDFs) of each group intersect, then it is possible to find a monotonic transformation of the ordinal scale that will change which group has a larger mean.

The second condition introduces the concept of the line of independence minus absolute concentration (LMA) curve. As the name suggests, the LMA curve takes the difference between two curves: the line of independence (LoI) and the absolution concentration curve. The LoI is defined as the weighted mean of the dependent variable, Y, multiplied by the cumulative distribution, $F(X)$ , of the explanatory variable, X:

(3) $$ \begin{align} \textit{LoI}_{i} = \big[ \frac{1}{N} \sum^{N}_{i=1} w_{i} Y_{i} \big] \times F(X_{i}). \end{align} $$

The absolute concentration curve (ACC) is the cumulated product of the dependent variable, Y, and the frequency weight, w, divided by the sum of the frequency weights:

(4) $$ \begin{align} \textit{ACC}_{1} = \frac{Y_{1} w_{1}}{\sum^{N}_{i=1} w_{i}}, \ \ \ \ ACC_{2} = \frac{Y_{1} w_{1} + Y_{2} w_{2}}{\sum^{N}_{i=1} w_{i}}, ... \ \ , ACC_{N} = \frac{Y_{1} w_{1} + ... + Y_{N} w_{N}}{\sum^{N}_{i=1} w_{i}}. \end{align} $$

In both Equations (3) and (4), $Y_{1} \leq Y_{2} \leq \cdots \leq Y_{N}$ . Equation (4) can be interpreted as the generalized Lorenz curve. Finally, the LMA curve is defined as follows:Footnote ⁹

(5) $$ \begin{align} \textit{LMA}_{i} = LoI_{i} - ACC_{i} \ \ \ \ \ \ \ \ \forall \ \ N. \end{align} $$

The LMA curve is related to the concept of second-order stochastic dominance and the absolute Lorenz curve. Recall that second-order stochastic dominance states that if two Lorenz curves cross, then it is impossible to determine which of two mutually exclusive groups second-order stochastically dominates the other. Therefore, the second condition, derived by Schröder and Yitzhaki (Reference Schröder and Yitzhaki2017), states that if the LMA curve intersects the horizontal axis, then the absolute Lorenz curve intersect the LoI, and there is some monotonic increasing transformation that will change the sign of the OLS regression coefficient.Footnote ¹⁰ If the absolute Lorenz curves do not cross, then there does not exist a monotonic increasing transformation that can change the sign.Footnote ¹¹

Stated more formally, the logic of the second condition is as follows. Consider two simple bivariate OLS regression coefficients, $\alpha _{1}$ and $\beta _{1}$ , from two separate specifications. One uses the raw ordinal variable, Y, and the other uses the transformed ordinal variable, $T(Y)$ . If the LMA curve of Y, with respect to X, intersects the horizontal axis, it is possible to find a monotonic increasing transformation of the dependent variable, $T(Y)$ , that can change the sign of the OLS regression coefficient. That is, if $\alpha _{1}$ is positive (negative), then $\beta _{1}$ will be negative (positive). This implies:

(6) $$ \begin{align} \frac{\alpha_{1}}{\beta_{1}} = \frac{\frac{Cov(X,Y)}{Var(X)}}{\frac{Cov(X,T(Y))}{Var(X)}} < 0. \end{align} $$

Although these sufficient conditions are instructive, important questions remain. First, suppose there exists a transformation $T(Y)$ that allows Equation (6) to hold, what is the shape of this transformation? Second, on the other hand, suppose there does not exist a transformation $T(Y)$ that allows Equation (6) to hold, does the magnitude of the coefficient meaningfully change? Similarly, how is statistical significance affected by these transformations? Finally, Equation (1) displayed an analytical example where there are multiple covariates, and Equations (3)–(6) only consider one X variable. This raises a final question. Since the existing tests of Schröder and Yitzhaki (Reference Schröder and Yitzhaki2017) only focus on simple bivariate examples, how are researchers to test robustness of more complicated specifications to monotonic increasing transformations of the ordinal scale? These are the questions that this paper now aims to address.

3.1 Globally Concave and Convex Transformations

In the spirit of Oswald (Reference Oswald2008), the reporting function can be convex, concave, or linear in the raw ordinal rankings. The following parameterized function effectively allows the reporting function to be convex, concave, or linear depending on the value of the $\sigma $ parameter:

(7) $$ \begin{align} \textit{T({Y})} = Y_{Max} \times \left(\frac{\textit{Y}}{Y_{Max}}\right)^{\sigma} \ \ \ \ \ \ \ \ \ \ \ \ \forall \ \ \sigma> 0. \end{align} $$

In this transformation, Y is the linear ordinal scale ranging from zero to $Y_{Max}$ —where $Y_{Max}$ is the maximum value of the observed ordinal scale.Footnote ¹² If $\sigma = 1$ , then the scale remains in its linear form. If $0 < \sigma < 1$ , then the scale will be concave to some degree, with the distances between relatively low levels being larger than the distances between relatively high levels. Finally, if $\sigma> 1$ , then the ordinal scale will be convex to some degree, with the distances between relatively low levels being smaller than the distances between relatively high levels.

Theoretically, values of $\sigma $ could exist within the positive infinite interval $(0, +\infty )$ . If some restrictions to this domain are acceptable, however, Equation (7) can help provide insight into the robustness of a particular effect estimate to plausible monotonic increasing transformations. Figure 1 shows Equation (7), assuming—for illustrative purposes—a 0–10 ordinal scale, with several values of $\sigma $ . Plotting these functions allows researchers to make a choice about restrictions to the domain of transformations.

Figure 1 Specific parameter values of globally concave and convex transformations. Notes: This figure shows various transformation functions, given specific parameter values. The functions map the original variable, Y, into a transformed ordinal variable, T(Y). In this figure, the ordinal scale is assumed to run from 0 to 10.

As argued by Kaiser and Vendrik (Reference Kaiser and Vendrik2019), graphically assessing plausibility may be the best and most transparent approach. Alternative ways for assessing plausibility could include benchmarking the shape of the distribution of the ordinal variable based on some other variable that measures a related concept. This is the approach used by Bond and Lang (Reference Bond and Lang2019) who argue that the distribution of subjective well-being can be just as skewed as the wealth distribution. Although this method of benchmarking can be potentially informative, it is still widely debated by empirical researchers. Kaiser and Vendrik (Reference Kaiser and Vendrik2019) demonstrate that the transformations used by Bond and Lang (Reference Bond and Lang2019) imply unrealistic assumptions about the way individuals reply to survey questions. Specifically, although the log-normal transformation used by (Bond and Lang Reference Bond and Lang2019) is theoretically permissible, it implies that small changes at low levels of true unobservable happiness are able to change responses on the observed ordinal scale by several categories and—at the same time—only large changes at high levels of true unobservable happiness are able to change responses on the observed ordinal scale. Moreover, Kaiser and Vendrik (Reference Kaiser and Vendrik2019) point out that it is unclear how knowing the predicted distribution of happiness is more or less skewed than the income distribution informs about the plausibility of a monotonic increasing transformation. If higher incomes are subject to decreasing marginal utility—as is assumed by many researchers and (arguably) empirically supported—then we should expect the distribution of true unobservable happiness to be less skewed than the income distribution.

For the subsequent empirical illustrations, I assume that $\sigma \in [0.1, 10]$ define the range of transformations. In each of the empirical illustrations, this range of transformations is relatively extreme. In the analysis of trust, shown in the main text, a transformation at each end of this range implies massive changes in latent trust associated with either only relatively low or relatively high reported trust levels. This range of transformations is purposefully extreme. It allows the researcher to compute a large set of effect estimates associated with this wide range of transformations. As I will discuss in Section 3.3, the final step of this method is to assess a (more narrow) range of plausible transformations.

3.2 Transformations with an Inflection Point

An alternative class of transformations are those with an inflection point. Rather than being either concave or convex, this class of transformations are convex below and concave above an inflection point. The motivation for this class of transformations extends the intuition of Oswald (Reference Oswald2008), where people are reluctant to report the highest values of some ordinal scale. If the “true” reporting function is one with an inflection point, this would imply that people are also reluctant to report the lowest values on the scale. Said differently, it takes a relatively large marginal gain or loss of some latent variable to move an individual off the midpoint of the observed ordinal scale.

One way to define such a class of reporting functions is to transform the observed ordinal scale as follows:

(8) $$ \begin{align} \textit{T({Y})} = Y_{Max} \times F\Big(\frac{X-Y_{Mid}}{\sigma}\Big). \end{align} $$

In Equation (8), $F(\cdot )$ is the CDF with a mean of $Y_{Mid}$ —the middle point on the ordinal scale—and a standard deviation of $\sigma $ . The domain of $\sigma $ is determined by the range of the ordinal scale. In general, if the scale is zero through $Y_{Max}$ and $\sigma = Y_{Mid}$ , then $F(\cdot )$ will essentially be linear.Footnote ¹³ If $\sigma $ is relatively close to zero, then $F(\cdot )$ will look increasingly like a step function, with the step at $Y_{Mid}$ . Therefore, limiting the domain of $\sigma $ is straightforward when using these CDF transformations with an inflection point. Defining $\sigma \in [0.1, Y_{Mid}]$ effectively characterizes all transformations from the nearly linear case to the relatively extreme step-function case. These details are visualized in Figure 2 for a 0–10 scale.

Figure 2 Specific parameter values of CDF transformations. Notes: This figure shows various transformation functions, given specific parameter values. The functions map the original variable, Y, into a transformed ordinal variable, T(Y). In this figure, the scale is assumed to run from 0 to 10.

3.3 Assessing Plausibility

The final step of this method—which is only necessary if effect estimates are not robust to the initial range of transformations—is to limit the set of effect estimates to those associated with a range of plausible transformations. As the work of Kaiser and Vendrik (Reference Kaiser and Vendrik2019) demonstrates, this assessment is best and most transparent when implemented graphically. This is not to say that other ways for assessing plausibility are inappropriate. Alternative ways such as (i) finding the range of transformations that preserve empirical results that must be true based on theory or (ii) benchmarking the shape of the distribution of the ordinal variable based on some other variable that measures a related concept could be effective in the appropriate context.Footnote ¹⁴ The present goal, however, is to provide a framework for assessing plausibility that could be reasonably applied in most empirical settings.

Graphically assessing plausibility first requires the researcher to examine the set of effect estimates associated with the relatively extreme range of transformations. If the estimated effects are qualitatively robust for the entire range of transformations, then the researcher can conclude that estimated effects are robust to extreme monotonic increasing transformations. Conversely, if the estimated effects are not robust in sign, statistical significance, and magnitude for the entire range of (relatively extreme) transformations, then the researcher must assess a range of plausible transformations, given the empirical context. I demonstrate this sort of analysis in the subsequent empirical illustration.

This final step may seem esoteric. It is important to acknowledge, however, that it is generally not always possible to perform credible statistical inference without any untestable assumptions (see, e.g., Conley, Hansen, and Rossi Reference Conley, Hansen and Rossi2012). As discussed by Tamer (Reference Tamer2010, p. 168), “the partial identification approach to econometrics views economic models as sets of assumptions, some of which are plausible and some of which are esoteric and are needed only to complete a model.” In this sense, partial identification allows for a way to test the sensitivity of effect estimates to these seemingly esoteric assumptions. Stronger assumptions (e.g., assuming a linear reporting function associated with an ordinal variable) will naturally allow for more information about a given effect estimate, however, as the strength of the necessary assumptions increases, the credibility of the effect estimate decreases (Coombs Reference Coombs1965; Manski Reference Manski2003).

Any assessment of plausible monotonic increasing transformations will critically rely on the specific elements of a given empirical application. Much like, for example, plausibly exogenous instrumental variables (Conley et al. Reference Conley, Hansen and Rossi2012). In the empirical applications discussed in this paper, I rely on existing research that specifically assesses the plausibility of various transformations of subjective well-being and happiness scales (Banks and Coleman Reference Banks and Coleman1981; Van Praag Reference Van Praag1991; Oswald Reference Oswald2008; Kaiser and Vendrik Reference Kaiser and Vendrik2019) and the cardinal properties of standardized test scores (Reardon Reference Reardon2008; Nielsen Reference Nielsen2017). Specifically, and as discussed in more detail in Section 4.2, if the reporting function of subjective well-being is curved at all, it is likely to be concave and the relationship between test scores and student learning is likely convex. Combining the general analytical approach developed in this paper with specific assessments of plausible transformations within a given empirical context provides useful structure for testing robustness of the cardinal treatment of ordinal variables.

4.1 Graphical Results

Figures 4 and 5 show effect estimates for each of the two classes of transformations, for each of the five measures of trust in others, relating to the results reported in Table 5 of Nunn and Wantchekon (Reference Nunn and Wantchekon2011). The authors argue that the magnitudes of these estimated effects are economically meaningful. Specifically, a one standard deviation change in the intensity of the slave trade represents a $-$ 1.10 to $-$ 0.16 standard deviation change in each of the five different measures of trust in others. The authors also perform a variance decomposition to assess economic significance, as discussed below.

Figure 4 Effect sets for Nunn and Wantchekon (Reference Nunn and Wantchekon2011)—globally concave and convex transformations. Notes: The dark lines represent the point estimates for a given specification with the corresponding value of log( $\sigma $ ). Logging the value of $\sigma $ allows for equal share of the graph to represent concave and convex transformations. Lighter lines represent 95% confidence interval calculated with standard errors clustered by ethnicity. Each panel refers to a different specification used in Table 5 of Nunn and Wantchekon (Reference Nunn and Wantchekon2011). Panel A refers to column (1) with the dependent variable trust of relatives. Panel B refers to column (2) with the dependent variable trust of neighbors. Panel C refers to column (3) with the dependent variable trust of local council. Panel D refers to column (4) with the dependent variable intragroup trust. Finally, Panel E refers to column (5) with the dependent variable intergroup trust.

Figure 5 Effect sets for Nunn and Wantchekon (Reference Nunn and Wantchekon2011)—transformations with an inflection point. Notes: The dark lines represent the point estimates for a given specification with the corresponding sigma value. Lighter lines represent 95% confidence interval calculated with standard errors clustered by ethnicity. Each panel refers to a different specification used in Table 5 of Nunn and Wantchekon (Reference Nunn and Wantchekon2011), presenting instrumental variable estimation results. Panel A refers to column (1) with the dependent variable trust of relatives. Panel B refers to column (2) with the dependent variable trust of neighbors. Panel C refers to column (3) with the dependent variable trust of local council. Panel D refers to column (4) with the dependent variable intragroup trust. Finally, Panel E refers to column (5) with the dependent variable intergroup trust.

4.2 Plausible Effect Bounds

So far, the range of alternative transformations could be characterized as relatively extreme. In the case of the globally concave and convex transformations, the range is defined by any $\sigma \in [0.1, 10]$ . A transformation associated with $\sigma =0.1$ implies that moving from a reported trust score of 1—on the 0–3 ordinal scale—measures a massive relative change in latent trust, while a transformation associated with $\sigma =10$ implies only moving from a reported trust score of 3 to a score of 4 measures any noticeable relative change in latent trust. In the case of transformations with an inflection point, the range extends from transformations suggesting a step function to transformations that are roughly linear. Step-function transformations imply that the only useful information embedded within the trust score relating to latent trust is whether or not the respondent reports a trust score of greater than 2. A roughly linear transformation, on the other hand, implies that each trust score represents an equal marginal gain in latent trust. In both of these cases, the range of alternative monotonic increasing transformations likely represent transformations that are implausible in this specific empirical setting.

As noted above, however, the estimated effect sets reported in Figures 4 and 5 show that the results of Nunn and Wantchekon (Reference Nunn and Wantchekon2011) are robust to these relatively extreme and likely implausible transformations. This finding of robustness to extreme transformations provides evidence supporting the cardinal treatment of ordinal variables.Footnote ¹⁶

This method requires one additional step in applications where estimated effects are not robust to a relatively extreme set of transformations—as is the case when reanalyzing the results from Aghion et al. (Reference Aghion, Akcigit, Deaton and Roulet2016) and Bond and Lang (Reference Bond and Lang2013), shown in the Online Supplement. The task then is to determine a plausible range of transformations and, therefore, plausible bounds on the estimated effects.

What is a plausible transformation of the ordinal scale in these empirical contexts? In the case of Aghion et al. (Reference Aghion, Akcigit, Deaton and Roulet2016), the work of Kaiser and Vendrik (Reference Kaiser and Vendrik2019), who focus on assessing the plausibility of transformations to subjective well-being and happiness scales, is instructive. Kaiser and Vendrik (Reference Kaiser and Vendrik2019) cite three studies that experimentally aim to identify the functional form of the reporting function defining the relationship between subjective feelings and objective reality. Empirical analysis reported by Oswald (Reference Oswald2008) cannot reject that the reporting function is linear—if it is curved at all, it is slightly concave. Additionally, results found by Van Praag (Reference Van Praag1991) and Banks and Coleman (Reference Banks and Coleman1981) cannot rule out the finding that individuals in their study use a linear reporting function on average. Although these cited studies provide some suggestive evidence that assuming a linear reporting function may indeed be valid, this likely will be considered a relatively strong assumption. Assuming linearity of the reporting function leads to more precise effect estimates, but ultimately with less credibility.

Based on this assessment, Table A1 in the Online Supplement reports plausible bounds on the effect estimates for the results of Aghion et al. (Reference Aghion, Akcigit, Deaton and Roulet2016) based on transformations associated with $\sigma \in [0.4, 2.5]$ . This range of plausible transformations allows for both concave and convex transformations, but only transformations such that response categories are only 10 times larger on opposite ends of the scale.Footnote ¹⁷ As shown in the Online Supplement, when limiting alternative transformations to this more limited plausible range, I find that the set of effects extend from a small and statistically insignificant effect to an effect that is statistically significant and almost 50% larger than that originally reported. Thus, the results of Aghion et al. (Reference Aghion, Akcigit, Deaton and Roulet2016) are not robust to even plausible transformations of the ordinal scale measuring subjective well-being.

In the case of Bond and Lang (Reference Bond and Lang2013), the work of Reardon (Reference Reardon2008), and specifically insights from applying IRT results to the Early Childhood Longitudinal Study-Kindergarten (ECLS-K) data, provide structure for an assessment of plausible transformations. In particular, Reardon (Reference Reardon2008) estimates IRT parameters indicating how each question on the ECLS-K test predicts student learning.Footnote ¹⁸ The author finds that roughly 40% of the questions in the ECLS-K have a relatively high likelihood of predicting no information about student learning. Most fundamentally, this suggests that the relationship between the observed test score and student learning is unlikely to be linear. It also suggests that at low (high) test score levels, the marginal gain in student learning is relatively small (high). Taken together, this suggests that transformations with an inflection point are less plausible in the context of test scores, and implies that a plausible reporting function is convex to some degree. To what degree? Assuming that 40% of the questions could be answered correctly by guessing suggests that a $\sigma $ value of 5 is relatively plausible.Footnote ¹⁹ Therefore, a plausible range of transformations extends from $\sigma \in [1, 5]$ . As shown in the Online Supplement, when limiting alternative transformations to this more limited plausible range, I find the set of estimates largely replicates the core findings of Bond and Lang (Reference Bond and Lang2013) and support the fragility of these results to transformations of the test score scale.

This discussion demonstrates how this method can be used to test for robustness of effect estimates to the cardinal treatment of ordinal variables. As the supplemental illustrations shown in the Online Supplement highlight, this method can apply to virtually any variable that cannot be directly observed and must be quantitatively measured on an ordinal scale.