Kevin Murphy’s (Reference Murphy2021) focal article argues that “we [industrial-organizational (I-O) psychologists] should and can make much better use of simple statistics in arguing for our hypotheses, and a great place to start is Table 1 … because if N is either very large or very small, the outcomes of significance tests are a foregone conclusion and the only useful tool for making sense of results will be the descriptive statistics presented in Table 1”. The information that is typically reported in Table 1, such as means, standard deviations, reliabilities, and intercorrelations of study variables, allows us to sensibly understand the meaning of the data that are collected for a study and prevents the possibility of serious misinterpretation. Thus, in his view, “Table 1 is the Cinderella of tables”. Although Iagree with many points in this focal article including the points mentioned above, there are three issues on which Iwould like to comment.
First, as Murphy (Reference Murphy2021) lamented in the focal article, many researchers in I-O psychology provide yet ignore useful information (descriptive statistics) in Table 1. However, this does not necessarily mean that such information is buried in the literature. In fact, many I-O psychologists who conduct meta-analyses pay close attention to the information in Table 1, as demonstrated by the increasing use of meta-analysis as a vital tool for promoting cumulative science and evidence-based practice in I-O psychology and beyond. Most meta-analyses in the field synthesize either correlations or standardized mean differences that are reported and can be computed using the information found in Table 1. These meta-analytic results provide more accurate information about study variables’ bivariate associations and differences than any input studies by statistically synthesizing correlations or standardized mean differences across a number of input studies, thereby greatly reducing sampling error variance (or increasing overall sample sizes). As illustrated in many studies (e.g., Schmidt, Reference Schmidt2010), the correlations that are reported in typical studies in I-O psychology often have large sampling errors. In other words, we often see very different correlations across studies even for the same relationship assessed with the same measures, and this is in large part due to sampling error variance. That is, the correlations that are presented in Table 1 are often quite inaccurate (either upwardly or downwardly biased), leading to incorrect conclusions if they are interpreted by using significance testing. Of course, correlations from large sample-based studies are quite accurate given the relatively small amount of sampling error, but they are still less accurate than meta-analytic correlations and cannot provide such useful information as true heterogeneity (i.e., true between-studies variance available in every random-effects meta-analysis). That is, correlations and other descriptive statistics in Table 1, when taken at face value, can “look you in the eye and lie to you—without even blinking” (Schmidt, Reference Schmidt2010, p. 239) and thus should be interpreted with caution.Footnote 1
Second, Murphy (Reference Murphy2021) made a compelling case that many I-O psychologists “provide and then largely ignore Table 1,” but he failed to discuss whether the information that is typically reported in Table 1 is sufficient and/or what else should be included. An interesting yet odd observation Ihave made over the years (also shared by other researchers) is that most, if not all, studies include gender and age among demographic diversity-related variables as a study or control variable. However, ethnicity is rarely included despite the fact that most survey questionnaires include an item about ethnicity. To advance cumulative science in diversity and many other research fields in I-O psychology (e.g., personnel selection), it is important that studies sampling different ethnic groups, regardless of the percentage of minority respondents, include such information in Table 1. This information not only discloses whether a certain study under- or oversampled minority respondents but also facilitates future meta-analyses on ethnic group differences (e.g., whether minority employees experience incivility more or less frequently) or the moderating role of ethnicity (e.g., whether the negative relationship between incivility and job performance is stronger or weaker among minority employees).
Furthermore, as noted in the focal article, the statistical power of tests of nonlinear effects in I-O psychology is often low, leading to false conclusions, particularly when the test results are interpreted by using significance testing. However, meta-analysis “can still undo the damage done by erroneous data interpretation in individual studies, but only if the individual studies include the information needed to compute the effect sizes and make the needed corrections—not all do” (Schmidt, Reference Schmidt2010, p. 239). Therefore, it is important to include interaction and/or quadratic terms in Table 1 if nonlinear effects (e.g., interactive, curvilinear, and congruent effects) are examined. Such information will not only help to enhance transparency and promote open science but also facilitate future meta-analyses on nonlinear effects. (It is a misconception that meta-analysis can examine only bivariate, linear relationships.)
For example, several recent meta-analyses have examined the interactive effect of ability and motivation on performance (Van Iddekinge et al., Reference Van Iddekinge, Aguinis, Mackey and DeOrtentiis2018), the interactive effect of job demands and resources on strain (Gonzalez-Mulé et al., Reference Gonzalez-Mulé, Kim and Ryu2020), and the curvilinear relationship between abusive leadership and performance (Mackey et al., Reference Mackey, McAllister, Maher and Wang2019). The authors of these meta-analyses had to contact the first and/or corresponding author of almost every input study to request the information that was necessary to conduct such analyses given the lack of relevant information in Table 1 (e.g., the correlations of the multiplicative term between ability and motivation with ability, motivation, and performance in the case of Van Iddekinge et al.’s meta-analysis).
Additionally, studies that examine low-frequency or novel phenomena (e.g., employee theft, mortality) should include skewness and kurtosis in Table 1 and/or mention them in the method section as Murphy (Reference Murphy2021) suggested. Although it is not always necessary, this information will help readers better gauge the robustness of correlation-based advanced analyses (e.g., regression, hierarchical linear modeling) because such analyses usually require particular distributional assumptions. Of course, this distributional information can alternatively be provided as a figure—because a picture is worth a thousand words.
Third, Murphy’s (Reference Murphy2021) call for more attention to the descriptive statistics reported in Table 1 is timely and important but not new. Neal Schmitt, in his 1989 editorial as an incoming editor of the Journal of Applied Psychology, wrote, “I believe that many times simpler, more familiar data-analytic techniques would result in better communication of the findings of a study. Furthermore, these simpler procedures may, in fact, be much more appropriate” (p. 844). This was echoed in Philip Bobko’s Reference Bobko1995 editorial as an incoming editor of the Journal of Applied Psychology:
Please look at “simple” statistics, such as means, standard deviations, correlations, effect sizes, and so forth. And do not just look at them; consider them when attempting to understand and explain what is going on. Ibelieve that one can often (usually?) learn more by looking at these simple statistics with a critical and understanding eye than one can learn by computing the newest fashion in statistics with an amazed eye (p. 4).
To be clear, my purpose here is not saying the focal article’s points are overdue but emphasizing that it is time to act upon them and give due credit to Table 1.
In conclusion, in this commentary Ishare several challenges and pitfalls of which we should be mindful when Cinderella-izing Table 1. The information in Table 1 should not be blindly trusted, given the typical [small] sample size that we see in most articles in I-O psychology and beyond. Furthermore, Table 1 should include additional information to advance cumulative and open science as well as evidence-based practice by means of replication and meta-analysis (Eden, Reference Eden2002). Finally, Murphy’s (Reference Murphy2021) focal article should not be viewed as just another editorial but an earnest call for immediate action!
You have
Access
Kevin Murphy’s (Reference Murphy2021) focal article argues that “we [industrial-organizational (I-O) psychologists] should and can make much better use of simple statistics in arguing for our hypotheses, and a great place to start is Table 1 … because if N is either very large or very small, the outcomes of significance tests are a foregone conclusion and the only useful tool for making sense of results will be the descriptive statistics presented in Table 1”. The information that is typically reported in Table 1, such as means, standard deviations, reliabilities, and intercorrelations of study variables, allows us to sensibly understand the meaning of the data that are collected for a study and prevents the possibility of serious misinterpretation. Thus, in his view, “Table 1 is the Cinderella of tables”. Although Iagree with many points in this focal article including the points mentioned above, there are three issues on which Iwould like to comment.
First, as Murphy (Reference Murphy2021) lamented in the focal article, many researchers in I-O psychology provide yet ignore useful information (descriptive statistics) in Table 1. However, this does not necessarily mean that such information is buried in the literature. In fact, many I-O psychologists who conduct meta-analyses pay close attention to the information in Table 1, as demonstrated by the increasing use of meta-analysis as a vital tool for promoting cumulative science and evidence-based practice in I-O psychology and beyond. Most meta-analyses in the field synthesize either correlations or standardized mean differences that are reported and can be computed using the information found in Table 1. These meta-analytic results provide more accurate information about study variables’ bivariate associations and differences than any input studies by statistically synthesizing correlations or standardized mean differences across a number of input studies, thereby greatly reducing sampling error variance (or increasing overall sample sizes). As illustrated in many studies (e.g., Schmidt, Reference Schmidt2010), the correlations that are reported in typical studies in I-O psychology often have large sampling errors. In other words, we often see very different correlations across studies even for the same relationship assessed with the same measures, and this is in large part due to sampling error variance. That is, the correlations that are presented in Table 1 are often quite inaccurate (either upwardly or downwardly biased), leading to incorrect conclusions if they are interpreted by using significance testing. Of course, correlations from large sample-based studies are quite accurate given the relatively small amount of sampling error, but they are still less accurate than meta-analytic correlations and cannot provide such useful information as true heterogeneity (i.e., true between-studies variance available in every random-effects meta-analysis). That is, correlations and other descriptive statistics in Table 1, when taken at face value, can “look you in the eye and lie to you—without even blinking” (Schmidt, Reference Schmidt2010, p. 239) and thus should be interpreted with caution.Footnote 1
Second, Murphy (Reference Murphy2021) made a compelling case that many I-O psychologists “provide and then largely ignore Table 1,” but he failed to discuss whether the information that is typically reported in Table 1 is sufficient and/or what else should be included. An interesting yet odd observation Ihave made over the years (also shared by other researchers) is that most, if not all, studies include gender and age among demographic diversity-related variables as a study or control variable. However, ethnicity is rarely included despite the fact that most survey questionnaires include an item about ethnicity. To advance cumulative science in diversity and many other research fields in I-O psychology (e.g., personnel selection), it is important that studies sampling different ethnic groups, regardless of the percentage of minority respondents, include such information in Table 1. This information not only discloses whether a certain study under- or oversampled minority respondents but also facilitates future meta-analyses on ethnic group differences (e.g., whether minority employees experience incivility more or less frequently) or the moderating role of ethnicity (e.g., whether the negative relationship between incivility and job performance is stronger or weaker among minority employees).
Furthermore, as noted in the focal article, the statistical power of tests of nonlinear effects in I-O psychology is often low, leading to false conclusions, particularly when the test results are interpreted by using significance testing. However, meta-analysis “can still undo the damage done by erroneous data interpretation in individual studies, but only if the individual studies include the information needed to compute the effect sizes and make the needed corrections—not all do” (Schmidt, Reference Schmidt2010, p. 239). Therefore, it is important to include interaction and/or quadratic terms in Table 1 if nonlinear effects (e.g., interactive, curvilinear, and congruent effects) are examined. Such information will not only help to enhance transparency and promote open science but also facilitate future meta-analyses on nonlinear effects. (It is a misconception that meta-analysis can examine only bivariate, linear relationships.)
For example, several recent meta-analyses have examined the interactive effect of ability and motivation on performance (Van Iddekinge et al., Reference Van Iddekinge, Aguinis, Mackey and DeOrtentiis2018), the interactive effect of job demands and resources on strain (Gonzalez-Mulé et al., Reference Gonzalez-Mulé, Kim and Ryu2020), and the curvilinear relationship between abusive leadership and performance (Mackey et al., Reference Mackey, McAllister, Maher and Wang2019). The authors of these meta-analyses had to contact the first and/or corresponding author of almost every input study to request the information that was necessary to conduct such analyses given the lack of relevant information in Table 1 (e.g., the correlations of the multiplicative term between ability and motivation with ability, motivation, and performance in the case of Van Iddekinge et al.’s meta-analysis).
Additionally, studies that examine low-frequency or novel phenomena (e.g., employee theft, mortality) should include skewness and kurtosis in Table 1 and/or mention them in the method section as Murphy (Reference Murphy2021) suggested. Although it is not always necessary, this information will help readers better gauge the robustness of correlation-based advanced analyses (e.g., regression, hierarchical linear modeling) because such analyses usually require particular distributional assumptions. Of course, this distributional information can alternatively be provided as a figure—because a picture is worth a thousand words.
Third, Murphy’s (Reference Murphy2021) call for more attention to the descriptive statistics reported in Table 1 is timely and important but not new. Neal Schmitt, in his 1989 editorial as an incoming editor of the Journal of Applied Psychology, wrote, “I believe that many times simpler, more familiar data-analytic techniques would result in better communication of the findings of a study. Furthermore, these simpler procedures may, in fact, be much more appropriate” (p. 844). This was echoed in Philip Bobko’s Reference Bobko1995 editorial as an incoming editor of the Journal of Applied Psychology:
Please look at “simple” statistics, such as means, standard deviations, correlations, effect sizes, and so forth. And do not just look at them; consider them when attempting to understand and explain what is going on. Ibelieve that one can often (usually?) learn more by looking at these simple statistics with a critical and understanding eye than one can learn by computing the newest fashion in statistics with an amazed eye (p. 4).
To be clear, my purpose here is not saying the focal article’s points are overdue but emphasizing that it is time to act upon them and give due credit to Table 1.
In conclusion, in this commentary Ishare several challenges and pitfalls of which we should be mindful when Cinderella-izing Table 1. The information in Table 1 should not be blindly trusted, given the typical [small] sample size that we see in most articles in I-O psychology and beyond. Furthermore, Table 1 should include additional information to advance cumulative and open science as well as evidence-based practice by means of replication and meta-analysis (Eden, Reference Eden2002). Finally, Murphy’s (Reference Murphy2021) focal article should not be viewed as just another editorial but an earnest call for immediate action!