In the focal article, Ree, Carretta, and Teachout (Reference Ree, Carretta and Teachout2015) address a common error in research methods, in which researchers neglect the shared variance between facets of a multidimensional construct. We agree with the need to attend to the entire factor structure of constructs when using measures, whether in research or application. The objective of this commentary is to elaborate on useful practices when a dominant general factor (DGF), as defined by the focal article, is found to be present and, in particular, to explore cases of DGF results under research paradigms not considered by the focal article.
We hold, as a general principle, that particular data do not demand a particular analysis. Rather, particular data support many types of analyses. A method of analysis should be chosen so as to answer the research question, using only the data available and/or the data that can reasonably be collected.
That said, we agree that there is wisdom in doing an exploratory factor analysis or at least examining the matrix of correlations between scales and measures that are expected to constitute a multidimensional construct space. It may be the case that the scales are clearly differentiated by theory or by previous empirical studies; nevertheless, independence should not be assumed.
Beyond those general words, the specifics of the research context and the research question become relevant. In the context of test development, for example, application of a common factor model can be seen as a manipulation check: You get out of it what you put into it. McDonald (Reference McDonald1999) observes that it is “reasonable to suggest that any dimensional structure in cognition, personality, and so on, ‘discovered’ by factor analysis is there to discover because it was put there by the tests psychologists choose to invent” (p. 167). Both McDonald (Reference McDonald1999) and Carroll (Reference Carroll1993) observe that this can be of value, if one wants to assure oneself of the structure of the scales developed, individually and jointly. A test designed to provide a single score on a single scale should have a strong single factor, even if it is composed of multiple internal divisions: essential unidimensionality (Stout, Reference Stout1990). By contrast, a broad survey designed for internal comparisons, such as a job analysis survey, should show multiple well-measured dimensions on which individuals (or jobs) may be distinguished. Often, the reality is in between.
Another way to look at the same distinction is to consider levels of analysis, as defined by the research question but constrained by the data available. If the research question is broad, the highest level of analysis, and the DGF, may be appropriate. On the other hand, some research questions may be well addressed by an oblique simple structure, provided the correlation matrix is published (McDonald, Reference McDonald1999), or by first-order factors within a higher order factor model. As stated in the focal article, “Campbell, McCloy, Oppler, and Sager (Reference Campbell, McCloy, Oppler, Sager, Schmidt and Borman1993) described these as the highest order factors that could be useful” (emphasis added). That statement could be true even with a higher order factor acknowledged to be present; it might not be useful for the purpose of comparing or predicting job performance across different jobs.
These comments apply without regard to whether the method used is principal components analysis, a common factor model, or a modern cousin such as multidimensional item response theory.
Suppose one has a dataset in which the first orthogonal factor explains 50% of the variance, according to any appropriate method. What then? Is the glass (of explained variance) half empty or half full? The answer and the best approach depend considerably on one's context and intended application. We consider four cases.
1. Unintended g. An observed factor structure may be viewed as having an unintended general factor dominating multiple theoretically distinct scales: a DGF. As explored in the focal article, a DGF can be theoretically important, such as a higher order factor, but it may also be accurately described as nuisance g. Nuisance g can appear when a researcher intends to distinguish meaningful dimensions within a measurement space, such as in basic research or job analysis, and when the first factor is best explained as methodological (e.g., halo, impression management, or response set); these cases are well covered in the focal article.
2. Unintended s. A second case is that of unintended local dependence, marring a theoretically unidimensional construct (nuisance s). Such unintended, and unwanted, multidimensionality may occur when trying to distinguish individuals on a semantically unitary construct, for example, to make a selection or compensation decision or to assign a credential. Passage-based or scenario-based testlets (Wainer & Lewis, Reference Wainer and Lewis1990) often manifest factors that are not particularly useful to test designers but that should be accounted for in scoring. The management of unintended s is well described in the literature on test development (see, e.g., Stout, Reference Stout1990) and does not require extensive discussion here.
3. All factors intentional but used separately. The more interesting third and fourth cases are hybrids. In some settings, both first-order factors and a higher order factor may be relevant in a two-stage or n-stage analysis. Neither one is undesirable. For example, in testing, an overall score (higher order factor score) used to make a decision may be supplemented by diagnostic subscores (correlated factor scores). Subscores are expected to have an overall score component but also to provide some unique information; they are vector sums of a general and specific factor score.
Similarly, skills profiles in employee development may provide scores for distinct but not independent skills, which are individually meaningful to managers; for example, written and oral communication skills could be positively correlated and yet still developed through different means. At the same time, an overall assessment of skill or performance level could be useful to the same manager in making a holistic evaluation of progress.
It remains appropriate to analyze and consider a factor model encompassing all measures of the construct space. Correlated subscores need not be calculated in isolation; they can “borrow strength” from each other to improve on their individual reliability (Tukey, Reference Tukey1973). On the other hand, prior to reporting a subscore, it is important to verify that the subscore is adequately reliable for reporting and that enough of it is due to the independent component for it to add value to the overall score; often it is not (Reise, Moore, & Haviland, Reference Reise, Moore and Haviland2010; Sinharay & Puhan, Reference Sinharay and Puhan2007).
In cases where overall scores and subscores are desirable, it may prove useful to switch representations between use cases. A bifactor model can be used for evaluating the scale of the general and specific factors and, relatedly, the value added by the specific factors. Reise et al. (Reference Reise, Moore and Haviland2010) recommends adapting the explained common variance statistic to evaluate the dominance of the general factor; it is equally possible to calculate the statistic for specific factors in comparing their relative explanatory power. On the other hand, scores may be more interpretable by their end users if presented and labeled according to a correlated-factors model. Fortunately, mathematically speaking, correlated-factors models, or oblique simple structure models, are closely approximated by higher order factor models, and those are in turn equivalent to bifactor models such as the testlet response model (Thissen, Reference Thissen, Millsap, van der Ark, Bolt and Woods2013). The model and method used should provide a “syntactic match” to the research (or practice) question.
4. All factors intentional and used as a whole. Finally, in some cases, the entire construct space, in its full dimensionality, may be used as a unit in such a way that presentation is not a factor. For example, a complete set of measures may be provided as inputs to a regression or machine learning model. Provided sufficient data are available that a complex model can be evaluated, the general and specific factors may be used in an arbitrary configuration or in totality to predict an external variable.
Regression is better supported if correlations are not high, due to sparsely populated regions or, in the extreme, multicollinearity. However, small, well-measured correlations can be managed mathematically. Thus, orthogonal bifactor or full-factor models may perform better as inputs than do factor models with oblique axes, but oblique simple structure models are not ruled out entirely.
The presence of several clearly differentiated, reliable factor scores allows for redirection or rebalancing of an overall score away from the particular direction and composition of an “opportunistically composed” common factor. That is, initially, a DGF's meaning and utility are arbitrarily determined by the number, nature, and quality of the set of variables in which it was found (Carroll, Reference Carroll1993). A better composed variation can be determined if the specific factors are included in analysis. An example of this approach was given in the focal article: The core self-evaluation construct was labeled as an optimal weighting of four specific traits, where optimality was determined by validity against job satisfaction, job performance, and life satisfaction. In addition to validation contexts, optimal weighting-like approaches may also occur in cases of content balancing as mandated by a curriculum or job analysis.
Optimal weighting and similar approaches can be achieved with oblique factors but are likely easier to explore with orthogonal bifactor or full-factor models, for example, using regression.
In closing, we submit that there is not one single, simple approach to dealing with DGFs, applicable in all of the contexts in which they may appear. Instead, a set of complete and appropriate statistical methods—of which ignoring the largest source of variance is not one—can and should be considered according to their ability to answer the research or practical question at hand.
You have
Access
In the focal article, Ree, Carretta, and Teachout (Reference Ree, Carretta and Teachout2015) address a common error in research methods, in which researchers neglect the shared variance between facets of a multidimensional construct. We agree with the need to attend to the entire factor structure of constructs when using measures, whether in research or application. The objective of this commentary is to elaborate on useful practices when a dominant general factor (DGF), as defined by the focal article, is found to be present and, in particular, to explore cases of DGF results under research paradigms not considered by the focal article.
We hold, as a general principle, that particular data do not demand a particular analysis. Rather, particular data support many types of analyses. A method of analysis should be chosen so as to answer the research question, using only the data available and/or the data that can reasonably be collected.
That said, we agree that there is wisdom in doing an exploratory factor analysis or at least examining the matrix of correlations between scales and measures that are expected to constitute a multidimensional construct space. It may be the case that the scales are clearly differentiated by theory or by previous empirical studies; nevertheless, independence should not be assumed.
Beyond those general words, the specifics of the research context and the research question become relevant. In the context of test development, for example, application of a common factor model can be seen as a manipulation check: You get out of it what you put into it. McDonald (Reference McDonald1999) observes that it is “reasonable to suggest that any dimensional structure in cognition, personality, and so on, ‘discovered’ by factor analysis is there to discover because it was put there by the tests psychologists choose to invent” (p. 167). Both McDonald (Reference McDonald1999) and Carroll (Reference Carroll1993) observe that this can be of value, if one wants to assure oneself of the structure of the scales developed, individually and jointly. A test designed to provide a single score on a single scale should have a strong single factor, even if it is composed of multiple internal divisions: essential unidimensionality (Stout, Reference Stout1990). By contrast, a broad survey designed for internal comparisons, such as a job analysis survey, should show multiple well-measured dimensions on which individuals (or jobs) may be distinguished. Often, the reality is in between.
Another way to look at the same distinction is to consider levels of analysis, as defined by the research question but constrained by the data available. If the research question is broad, the highest level of analysis, and the DGF, may be appropriate. On the other hand, some research questions may be well addressed by an oblique simple structure, provided the correlation matrix is published (McDonald, Reference McDonald1999), or by first-order factors within a higher order factor model. As stated in the focal article, “Campbell, McCloy, Oppler, and Sager (Reference Campbell, McCloy, Oppler, Sager, Schmidt and Borman1993) described these as the highest order factors that could be useful” (emphasis added). That statement could be true even with a higher order factor acknowledged to be present; it might not be useful for the purpose of comparing or predicting job performance across different jobs.
These comments apply without regard to whether the method used is principal components analysis, a common factor model, or a modern cousin such as multidimensional item response theory.
Suppose one has a dataset in which the first orthogonal factor explains 50% of the variance, according to any appropriate method. What then? Is the glass (of explained variance) half empty or half full? The answer and the best approach depend considerably on one's context and intended application. We consider four cases.
1. Unintended g. An observed factor structure may be viewed as having an unintended general factor dominating multiple theoretically distinct scales: a DGF. As explored in the focal article, a DGF can be theoretically important, such as a higher order factor, but it may also be accurately described as nuisance g. Nuisance g can appear when a researcher intends to distinguish meaningful dimensions within a measurement space, such as in basic research or job analysis, and when the first factor is best explained as methodological (e.g., halo, impression management, or response set); these cases are well covered in the focal article.
2. Unintended s. A second case is that of unintended local dependence, marring a theoretically unidimensional construct (nuisance s). Such unintended, and unwanted, multidimensionality may occur when trying to distinguish individuals on a semantically unitary construct, for example, to make a selection or compensation decision or to assign a credential. Passage-based or scenario-based testlets (Wainer & Lewis, Reference Wainer and Lewis1990) often manifest factors that are not particularly useful to test designers but that should be accounted for in scoring. The management of unintended s is well described in the literature on test development (see, e.g., Stout, Reference Stout1990) and does not require extensive discussion here.
3. All factors intentional but used separately. The more interesting third and fourth cases are hybrids. In some settings, both first-order factors and a higher order factor may be relevant in a two-stage or n-stage analysis. Neither one is undesirable. For example, in testing, an overall score (higher order factor score) used to make a decision may be supplemented by diagnostic subscores (correlated factor scores). Subscores are expected to have an overall score component but also to provide some unique information; they are vector sums of a general and specific factor score.
Similarly, skills profiles in employee development may provide scores for distinct but not independent skills, which are individually meaningful to managers; for example, written and oral communication skills could be positively correlated and yet still developed through different means. At the same time, an overall assessment of skill or performance level could be useful to the same manager in making a holistic evaluation of progress.
It remains appropriate to analyze and consider a factor model encompassing all measures of the construct space. Correlated subscores need not be calculated in isolation; they can “borrow strength” from each other to improve on their individual reliability (Tukey, Reference Tukey1973). On the other hand, prior to reporting a subscore, it is important to verify that the subscore is adequately reliable for reporting and that enough of it is due to the independent component for it to add value to the overall score; often it is not (Reise, Moore, & Haviland, Reference Reise, Moore and Haviland2010; Sinharay & Puhan, Reference Sinharay and Puhan2007).
In cases where overall scores and subscores are desirable, it may prove useful to switch representations between use cases. A bifactor model can be used for evaluating the scale of the general and specific factors and, relatedly, the value added by the specific factors. Reise et al. (Reference Reise, Moore and Haviland2010) recommends adapting the explained common variance statistic to evaluate the dominance of the general factor; it is equally possible to calculate the statistic for specific factors in comparing their relative explanatory power. On the other hand, scores may be more interpretable by their end users if presented and labeled according to a correlated-factors model. Fortunately, mathematically speaking, correlated-factors models, or oblique simple structure models, are closely approximated by higher order factor models, and those are in turn equivalent to bifactor models such as the testlet response model (Thissen, Reference Thissen, Millsap, van der Ark, Bolt and Woods2013). The model and method used should provide a “syntactic match” to the research (or practice) question.
4. All factors intentional and used as a whole. Finally, in some cases, the entire construct space, in its full dimensionality, may be used as a unit in such a way that presentation is not a factor. For example, a complete set of measures may be provided as inputs to a regression or machine learning model. Provided sufficient data are available that a complex model can be evaluated, the general and specific factors may be used in an arbitrary configuration or in totality to predict an external variable.
Regression is better supported if correlations are not high, due to sparsely populated regions or, in the extreme, multicollinearity. However, small, well-measured correlations can be managed mathematically. Thus, orthogonal bifactor or full-factor models may perform better as inputs than do factor models with oblique axes, but oblique simple structure models are not ruled out entirely.
The presence of several clearly differentiated, reliable factor scores allows for redirection or rebalancing of an overall score away from the particular direction and composition of an “opportunistically composed” common factor. That is, initially, a DGF's meaning and utility are arbitrarily determined by the number, nature, and quality of the set of variables in which it was found (Carroll, Reference Carroll1993). A better composed variation can be determined if the specific factors are included in analysis. An example of this approach was given in the focal article: The core self-evaluation construct was labeled as an optimal weighting of four specific traits, where optimality was determined by validity against job satisfaction, job performance, and life satisfaction. In addition to validation contexts, optimal weighting-like approaches may also occur in cases of content balancing as mandated by a curriculum or job analysis.
Optimal weighting and similar approaches can be achieved with oblique factors but are likely easier to explore with orthogonal bifactor or full-factor models, for example, using regression.
In closing, we submit that there is not one single, simple approach to dealing with DGFs, applicable in all of the contexts in which they may appear. Instead, a set of complete and appropriate statistical methods—of which ignoring the largest source of variance is not one—can and should be considered according to their ability to answer the research or practical question at hand.