Normative Interpretation

Typically, meta-analysts evaluate the magnitude of heterogeneity in effect sizes by focusing on the face value of the true variability of the effect size distribution (SDρ) or the width of its associated credibility interval (cf. “the .30-unit width of the interval”; Tett, Hundley, & Christiansen, Reference Tett, Hundley and Christiansen2017, p. 8). This approach needs refinement, however, as correct interpretation of a credibility interval depends not only on its width but also on the range of correlations that it spans. Correlations in applied psychology are not uniformly distributed, and a number of reviews have summarized distributions of effect sizes (e.g., Bosco, Aguinis, Singh, Field, & Pierce, Reference Bosco, Aguinis, Singh, Field and Pierce2015; Gignac & Szodorai, Reference Gignac and Szodorai2016; Hemphill, Reference Hemphill2003). Among them, the most comprehensive review is that of Paterson, Harms, Steel, and Credé (Reference Paterson, Harms, Steel and Credé2016), which integrates 30 years of meta-analyses of micro-level variables in management to develop empirical benchmarks for observed and corrected correlations. An advantage of this approach is that, by constructing distributions from meta-analyses, upward biasing effects of sampling error on effect size variability are controlled.

Paterson et al. (Reference Paterson, Harms, Steel and Credé2016, p. 76) report a general distribution of corrected correlations with quartiles of .15, .25, and .39. According to these benchmarks, the distribution of correlations in applied psychology is positively skewed, with most correlations falling between .15 and .40. A consequence of this nonuniform distribution is that true variability is more substantively meaningful for some correlation ranges than for others. For example, a .20-unit wide interval from .08 to .28 reflects a range from comparatively negligible (<25th percentile of the empirical distribution) to moderate (between 50th and 75th percentiles) correlation values, whereas, the endpoints of a .20-unit wide interval from .55 to .75 would both be considered “very large” (>90th percentile). Variability is more meaningful, both for theoretical interpretations and practical applications (e.g., incremental validity), when it occurs near the center of the empirical distribution of effects.

An alternative to face-value interpretations of SDρ is comparing ρ and credibility interval endpoints to a relevant empirical distribution of effect sizes. Appropriate questions from this perspective include the following: What percentiles of the empirical distribution do the endpoints of the interval correspond to? Do endpoints reflect substantively different parts of the distribution, relative to the research question? What is the percentile distance between top and bottom bounds of the interval? If the credibility interval spans a wide range of the empirical distribution, then there may be value in searching for moderators. By comparison, when an interval covers a narrow slice of the empirical distribution, any potential moderators are unlikely to have a strong impact on theoretical interpretations or practical decisions.

Table 1 illustrates hypothetical combinations of ρ and SDρ values, along with corresponding credibility intervals, percentile values for Paterson et al.’s (Reference Paterson, Harms, Steel and Credé2016) overall empirical distribution for ρ, and the percentile distance between credibility interval endpoints. Several patterns are clear from these values. First, for ρ < .40, SDρ values between .07 and .11 translate to credibility intervals that cover major portions of the distribution of correlations found in industrial and organizational (I-O) psychology research. For larger correlations, however, even sizable SDρ values represent increasingly small slices of the distribution (e.g., endpoints of a credibility interval for ρ = .55, SDρ = .09 are only 21 percentile points apart). For ρ ≥ .60, both bounds of all credibility intervals examined fall in the upper 25% of applied effects. For effects this large, even ostensibly large variability is unlikely to alter substantive conclusions about the strength of relations between constructs.

Table 1. I-O Research Correlation Percentiles for Credibility Intervals

Note. Percentiles based on Paterson et al.’s (Reference Paterson, Harms, Steel and Credé2016) overall distribution of corrected correlations; bars in percentile columns reflect in which quarter of the correlation distribution values fall (white bars with border indicate that one bound of the interval crosses zero); percentile (%ile) distance range plots scaled to minimum of 0 and maximu m of 100; Cred. Int. = credibility interval; CV = credibility value.

Conversely, comparatively small variability can be substantively meaningful for some values of ρ. An SDρ = .03 is often regarded as negligible, and, indeed, for ρ ≤ .15 or ρ ≥ .35, this variability reflects a minor portion of the empirical distribution. However, for correlations between these values, this small SDρ corresponds to credibility intervals that span a wide section of the effect size distribution. Indeed, for a median correlation (ρ = .25), the credibility interval for SDρ = .03 spans 30% of empirical effects.

In sum, there is more substantive value for explaining variability and identifying potential moderators if the credibility interval covers a wide range of the empirical effect sizes. For SDρ values examined here, the largest meaningful variability occurs for |ρ| = .10 to .45. By contrast, variability for larger correlations will not affect central conclusions.

Table 2 presents the percentile ranges of credibility intervals for a sample of published meta-analyses reporting relations to Conscientiousness (based on Wilmot, Reference Wilmot2017). Consistent with the hypothetical credibility intervals in Table 1, the empirical credibility intervals cover the widest portion of the empirical effect size distribution (i.e., have the largest percentile difference) for correlations ranging from .10 to .30. Credibility intervals for larger correlations tend to cover less of the empirical distribution, even for relatively large SDρ values. For larger correlations, there is less value in exploring for moderators, as theoretical conclusions about the impact of Conscientiousness are unlikely to change for different values (e.g., whether Conscientiousness correlates ρ = .45 or .60 with Quality of Life, researchers would still conclude this relationship is very large). An extreme example is the relation of Conscientiousness to Grit. The correlation has a relatively large SDρ (.07), but the mean effect (ρ = .84) is so large that the entire credibility interval is above the 99th percentile of applied effects.Footnote ¹ Thus, moderator searches will be more meaningful for criteria with ρ located in the center of the distribution.

Table 2. Percentile Interpretations of Selected Published Credibility Intervals for Conscientiousness

Note. Table columns and color-coding correspond to those in Table 1; †reverse-coded; OCB = organizational citizenship behavior; CWB = counterproductive work behavior; all correlations corrected for unreliability in both variables; tabulated data based on Wilmot (Reference Wilmot2017).

Objective Interpretation

When dependent variables are expressed in units with meaningful, objective outcomes for individuals or organizations (e.g., salary, turnover rates, production costs), credibility intervals can also be interpreted in terms of their practical implications. For these criteria, variability can be translated into real-world consequences of correlations at each end of the interval. From this perspective, important questions include the following: Would applied decisions (e.g., to use a test in employee selection) change for the highest versus lowest interval values? Would conclusions about the importance of a variable for a real-world outcome change across the range of these correlations?

As an example, using the correlation with salary in Table 2 and the U. S. Census Bureau–reported SD of annual income for adults working ≥ 40 hours per week of $33,647, the impact of a 1 SD increase in Conscientiousness on individual salary varies from $7,066 for the upper bound value to negative $2,692 for the lower bound. This nearly $10,000 swing (and reversal of the effect's direction) clearly has practical implications for individuals and warrants investigation of moderators. Conversely, for turnover, assuming a base rate of 50% turnover and a selection ratio of .30, the lower and upper credibility values reflect a difference in reduced turnover rates of only 3%, which is less likely to substantively alter organizational decision making.