In their focal article, Ree, Carretta, and Teachout (Reference Ree, Carretta and Teachout2015) based their definition of a dominant general factor (DGF) on two criteria: (a) A DGF should be the largest source of reliable variance; (b) it is influencing every variable measuring the construct. Although detailed attention has been paid to the statistical properties of a DGF, I believe another criterion of equal if not greater importance is the theoretical justification to expect a DGF in the measurement of a construct. In the following commentary, I will highlight the importance of theory as another important criterion when determining the meaningfulness and usefulness of DGFs, discuss the risks of creating a DGF without any theoretical guidance, and elaborate on the complexities surrounding job performance as a detailed example to illustrate why theory is important before extracting a DGF from performance ratings.
Our theory of the construct will largely determine how we actually measure and model it (Hinkin, Reference Hinkin1998). The first risk of ignoring theory in identifying a DGF lies in the ambiguity of its substantive meaning. Assuming that one can find a DGF that affects every variable and explains the largest amount of variance, can the researcher know for sure that the DGF is a theoretically meaningful common factor? The answer is no unless we have a theory that points to the substantive meaning of the common factor. The two criteria given by the authors of the focal article are solely based on statistical results, with no reference to theory. Although they identified possible methodologicalFootnote
1
and theoretical reasons for the emergence of DGFs, this is a post hoc analysis. The purely data-driven process of identifying DGFs is contrary to good psychological measurement and might be even misleading in some cases. For example, if a researcher runs a factor analysis on family satisfaction, job satisfaction, and career satisfaction items, chances are there will be a DGF. But what is the theoretical underpinning of the DGF? Is it a higher-order construct called general life satisfaction, or is it also tapping into personality traits such as positive affectivity (PA)? Without a priori theoretical guidance, it is difficult to specify the substantive meaning of the DGF. What we intend to measure and what we know about the construct that we aim to measure should drive the measurement process, not vice versa. Having a theoretical grasp of the construct beforehand, researchers can also take steps to control for nuisances that might add ambiguity to the meaning of the DGF when designing the study. In the previous example, researchers can consider measuring PA as a covariate and see whether the DGF has any bearing on PA.
The second drawback of overlooking theory is the potential risk of erroneously specifying the measurement model. By simply subjecting items to different types of statistical analysis, researchers are, consciously or unconsciously, imposing their measurement model on the data. For example, if the causal direction is specified from the indicators to the construct, the construct is being treated as formative. Otherwise, it is considered to be reflective (Edwards & Bagozzi, Reference Edwards and Bagozzi2000). If the researcher is using principal component analysis (PCA), he or she is trying to do data reduction, whereas exploratory factor analysis (EFA) is appropriate for identifying common factors (Fabrigar, Wegener, MacCallum, & Strahan, Reference Fabrigar, Wegener, MacCallum and Strahan1999). Generally researchers will be able to correctly model the construct of interest provided that they have a good theoretical grasp of the construct (and statistical analysis of course). However, a problem arises if one limits his or her attention to the two statistical criteria while turning a blind eye to the nature of the construct. The two criteria given in the article are generally under the assumption that common factors underlie the covariance matrix in the first place. This assumption clearly does not apply to every construct. To illustrate this point, the model will be mistakenly specified if a researcher conducts EFA on items measuring socioeconomic status, a formative construct.
Further, in industrial–organizational psychology research, the exact nature of some constructs is more complex than simply being reflective or formative. One such construct is job performance, which was also used as an example in the focal article. However, prior to a detailed discussion of this construct, it is perhaps necessary to point out that among the three sources that the authors used to support the existence of a DGF in performance ratings, Lance, Teachout, and Donnelly (Reference Lance, Teachout and Donnelly1992) actually found that “performance is not a unitary construct” (p. 448). In their study, models that included a general performance factor fit significantly worse than the model with multiple performance dimensions. In another study that was cited (Carretta, Perry, & Ree, Reference Carretta, Perry and Ree1996), PCA was conducted on the supervisor and peer ratings, and the researchers identified a factor (i.e., a weighted linear composite of ratings from different sources) that could explain 92.5% of the variance. The primary objective of PCA is data reduction, which is different from the identification of latent constructs (Fabrigar et al., Reference Fabrigar, Wegener, MacCallum and Strahan1999). Citing this study to argue for the existence of general factors in job performance ratings could be considered as an incorrectly executed measurement model in its own.
Nonetheless, Viswesvaran, Schmidt, and Ones (Reference Viswesvaran, Schmidt and Ones2005), in their comprehensive meta-analysis of job performance ratings, found that a general factor in job performance ratings accounted for more than half of the variation independent of halo and other measurement errors. The authors attributed the general factor to the effects of organizational citizenship behaviors (OCBs) clouding performance ratings on all dimensions and basic individual-difference variables such as conscientiousness. As such, the antecedents to the general factor among job performance dimensions at least involve both personal attributes (e.g., conscientiousness) and situational variables (e.g., raters influenced by ratees’ citizenship behaviors; one can further argue this is a person–situation interaction). It thus suffices to say that the DGF in job performance ratings is more complex than in ability and personality domains that mainly involve individual characteristics (assuming that methodological artifacts are already controlled for, as in Viswesvaran et al.’s study). This is probably one of the reasons that Viswesvaran and colleagues called for theory development in job performance research to address this general factor. Therefore, it seems too early to talk about the implications of DGFs before researchers are able to gain a solid theoretical understanding of DGFs in job performance research.
In fact, Edwards (Reference Edwards2001) classified job performance as an aggregate construct, which is produced when all of its dimensions are added together (Law, Wong, & Mobley, Reference Law, Wong and Mobley1998; Murphy & Shiarella, Reference Murphy and Shiarella1997). This is congruent with Campbell, McCloy, Oppler, and Sager (Reference Campbell, McCloy, Oppler, Sager, Schmitt and Borman1993) regarding the emphasis on the distinct dimensions of job performance. As Viswesvaran and colleagues (Reference Viswesvaran, Schmidt and Ones2005) aptly pointed out, this approach does not run counter to the general factor in job performance ratings.Footnote
2
Either one (focusing on the general factor vs. specific factors) has its value (for a detailed discussion and examples see Viswesvaran et al., Reference Viswesvaran, Schmidt and Ones2005). Furthermore, it is equally important to correctly model the relationships among performance dimensions even in the absence of a general factor. For example, Bergeron, Shipp, Rosen, and Furst (Reference Bergeron, Shipp, Rosen and Furst2013) found that time spent on OCB could be negatively related to task performance given the limited time that an individual can invest into different types of work behaviors. Wallace and Chen (Reference Wallace and Chen2006) found safety performance, a specific aspect of work behaviors, to be negatively related to production performance. As such, the existence of DGFs, or lack thereof, may both have complex theoretical reasons. Simply chasing DGFs in performance ratings without any consideration of the theoretical underpinnings runs the risk of missing the whole picture.
The discussion thus far is not intended to nullify the value of DGFs but to rather highlight the importance of theory as a prerequisite of DGFs. Without a priori theoretical understanding, DGFs might be at best interpreted as a general factor among the variables, its substantive meaning being unknown. Post hoc explanations are possible but will not be as helpful. The delicacy of this issue is further exemplified in the case of job performance. Again, the central thesis is that theory should be an important factor in determining the meaningfulness and usefulness of DGFs (assuming methodological artifacts are already accounted for). Although I have repeatedly referred to theory as the crux of the matter, it is by no means an easy task to have a clearly spelled-out theory for every construct. However, this should not be taken as an excuse to shy away from theory. If we abandon theory, we might get lost in chasing DGFs from random sets of variables in a pure data-driven manner.
To conclude this commentary, I would like to use the same example from the focal article. In validating the construct of core self-evaluation (CSE), Judge, Locke, and Durham (Reference Judge, Locke and Durham1997) went great lengths to find theoretical guidance from clinical and social psychology (e.g., Packer, Reference Packer1985a, Reference Packer1985b) so that they can establish the theoretical legitimacy of CSE. The attention to theory prior to the identification of DGFs (in this case, CSE) is equally, if not more, important as the quantitative validation that was cited (e.g., Judge, Erez, Bono, & Thoresen, Reference Judge, Erez, Bono and Thoresen2003). Although discovering a DGF is tempting, it will be nothing more than an ambiguous general factor unless we have a good theoretical background to derive substantive meaning out of it.
In their focal article, Ree, Carretta, and Teachout (Reference Ree, Carretta and Teachout2015) based their definition of a dominant general factor (DGF) on two criteria: (a) A DGF should be the largest source of reliable variance; (b) it is influencing every variable measuring the construct. Although detailed attention has been paid to the statistical properties of a DGF, I believe another criterion of equal if not greater importance is the theoretical justification to expect a DGF in the measurement of a construct. In the following commentary, I will highlight the importance of theory as another important criterion when determining the meaningfulness and usefulness of DGFs, discuss the risks of creating a DGF without any theoretical guidance, and elaborate on the complexities surrounding job performance as a detailed example to illustrate why theory is important before extracting a DGF from performance ratings.
Our theory of the construct will largely determine how we actually measure and model it (Hinkin, Reference Hinkin1998). The first risk of ignoring theory in identifying a DGF lies in the ambiguity of its substantive meaning. Assuming that one can find a DGF that affects every variable and explains the largest amount of variance, can the researcher know for sure that the DGF is a theoretically meaningful common factor? The answer is no unless we have a theory that points to the substantive meaning of the common factor. The two criteria given by the authors of the focal article are solely based on statistical results, with no reference to theory. Although they identified possible methodologicalFootnote 1 and theoretical reasons for the emergence of DGFs, this is a post hoc analysis. The purely data-driven process of identifying DGFs is contrary to good psychological measurement and might be even misleading in some cases. For example, if a researcher runs a factor analysis on family satisfaction, job satisfaction, and career satisfaction items, chances are there will be a DGF. But what is the theoretical underpinning of the DGF? Is it a higher-order construct called general life satisfaction, or is it also tapping into personality traits such as positive affectivity (PA)? Without a priori theoretical guidance, it is difficult to specify the substantive meaning of the DGF. What we intend to measure and what we know about the construct that we aim to measure should drive the measurement process, not vice versa. Having a theoretical grasp of the construct beforehand, researchers can also take steps to control for nuisances that might add ambiguity to the meaning of the DGF when designing the study. In the previous example, researchers can consider measuring PA as a covariate and see whether the DGF has any bearing on PA.
The second drawback of overlooking theory is the potential risk of erroneously specifying the measurement model. By simply subjecting items to different types of statistical analysis, researchers are, consciously or unconsciously, imposing their measurement model on the data. For example, if the causal direction is specified from the indicators to the construct, the construct is being treated as formative. Otherwise, it is considered to be reflective (Edwards & Bagozzi, Reference Edwards and Bagozzi2000). If the researcher is using principal component analysis (PCA), he or she is trying to do data reduction, whereas exploratory factor analysis (EFA) is appropriate for identifying common factors (Fabrigar, Wegener, MacCallum, & Strahan, Reference Fabrigar, Wegener, MacCallum and Strahan1999). Generally researchers will be able to correctly model the construct of interest provided that they have a good theoretical grasp of the construct (and statistical analysis of course). However, a problem arises if one limits his or her attention to the two statistical criteria while turning a blind eye to the nature of the construct. The two criteria given in the article are generally under the assumption that common factors underlie the covariance matrix in the first place. This assumption clearly does not apply to every construct. To illustrate this point, the model will be mistakenly specified if a researcher conducts EFA on items measuring socioeconomic status, a formative construct.
Further, in industrial–organizational psychology research, the exact nature of some constructs is more complex than simply being reflective or formative. One such construct is job performance, which was also used as an example in the focal article. However, prior to a detailed discussion of this construct, it is perhaps necessary to point out that among the three sources that the authors used to support the existence of a DGF in performance ratings, Lance, Teachout, and Donnelly (Reference Lance, Teachout and Donnelly1992) actually found that “performance is not a unitary construct” (p. 448). In their study, models that included a general performance factor fit significantly worse than the model with multiple performance dimensions. In another study that was cited (Carretta, Perry, & Ree, Reference Carretta, Perry and Ree1996), PCA was conducted on the supervisor and peer ratings, and the researchers identified a factor (i.e., a weighted linear composite of ratings from different sources) that could explain 92.5% of the variance. The primary objective of PCA is data reduction, which is different from the identification of latent constructs (Fabrigar et al., Reference Fabrigar, Wegener, MacCallum and Strahan1999). Citing this study to argue for the existence of general factors in job performance ratings could be considered as an incorrectly executed measurement model in its own.
Nonetheless, Viswesvaran, Schmidt, and Ones (Reference Viswesvaran, Schmidt and Ones2005), in their comprehensive meta-analysis of job performance ratings, found that a general factor in job performance ratings accounted for more than half of the variation independent of halo and other measurement errors. The authors attributed the general factor to the effects of organizational citizenship behaviors (OCBs) clouding performance ratings on all dimensions and basic individual-difference variables such as conscientiousness. As such, the antecedents to the general factor among job performance dimensions at least involve both personal attributes (e.g., conscientiousness) and situational variables (e.g., raters influenced by ratees’ citizenship behaviors; one can further argue this is a person–situation interaction). It thus suffices to say that the DGF in job performance ratings is more complex than in ability and personality domains that mainly involve individual characteristics (assuming that methodological artifacts are already controlled for, as in Viswesvaran et al.’s study). This is probably one of the reasons that Viswesvaran and colleagues called for theory development in job performance research to address this general factor. Therefore, it seems too early to talk about the implications of DGFs before researchers are able to gain a solid theoretical understanding of DGFs in job performance research.
In fact, Edwards (Reference Edwards2001) classified job performance as an aggregate construct, which is produced when all of its dimensions are added together (Law, Wong, & Mobley, Reference Law, Wong and Mobley1998; Murphy & Shiarella, Reference Murphy and Shiarella1997). This is congruent with Campbell, McCloy, Oppler, and Sager (Reference Campbell, McCloy, Oppler, Sager, Schmitt and Borman1993) regarding the emphasis on the distinct dimensions of job performance. As Viswesvaran and colleagues (Reference Viswesvaran, Schmidt and Ones2005) aptly pointed out, this approach does not run counter to the general factor in job performance ratings.Footnote 2 Either one (focusing on the general factor vs. specific factors) has its value (for a detailed discussion and examples see Viswesvaran et al., Reference Viswesvaran, Schmidt and Ones2005). Furthermore, it is equally important to correctly model the relationships among performance dimensions even in the absence of a general factor. For example, Bergeron, Shipp, Rosen, and Furst (Reference Bergeron, Shipp, Rosen and Furst2013) found that time spent on OCB could be negatively related to task performance given the limited time that an individual can invest into different types of work behaviors. Wallace and Chen (Reference Wallace and Chen2006) found safety performance, a specific aspect of work behaviors, to be negatively related to production performance. As such, the existence of DGFs, or lack thereof, may both have complex theoretical reasons. Simply chasing DGFs in performance ratings without any consideration of the theoretical underpinnings runs the risk of missing the whole picture.
The discussion thus far is not intended to nullify the value of DGFs but to rather highlight the importance of theory as a prerequisite of DGFs. Without a priori theoretical understanding, DGFs might be at best interpreted as a general factor among the variables, its substantive meaning being unknown. Post hoc explanations are possible but will not be as helpful. The delicacy of this issue is further exemplified in the case of job performance. Again, the central thesis is that theory should be an important factor in determining the meaningfulness and usefulness of DGFs (assuming methodological artifacts are already accounted for). Although I have repeatedly referred to theory as the crux of the matter, it is by no means an easy task to have a clearly spelled-out theory for every construct. However, this should not be taken as an excuse to shy away from theory. If we abandon theory, we might get lost in chasing DGFs from random sets of variables in a pure data-driven manner.
To conclude this commentary, I would like to use the same example from the focal article. In validating the construct of core self-evaluation (CSE), Judge, Locke, and Durham (Reference Judge, Locke and Durham1997) went great lengths to find theoretical guidance from clinical and social psychology (e.g., Packer, Reference Packer1985a, Reference Packer1985b) so that they can establish the theoretical legitimacy of CSE. The attention to theory prior to the identification of DGFs (in this case, CSE) is equally, if not more, important as the quantitative validation that was cited (e.g., Judge, Erez, Bono, & Thoresen, Reference Judge, Erez, Bono and Thoresen2003). Although discovering a DGF is tempting, it will be nothing more than an ambiguous general factor unless we have a good theoretical background to derive substantive meaning out of it.