Dimensional models of psychopathology have been shown to be superior to categorical (i.e. diagnostic) models in terms of identifying genetic and environmental risk (Taylor et al., Reference Taylor, Martin, Lu, Brikell, Lundström, Larsson and Lichtenstein2019), underlying neurology (Goodkind et al., Reference Goodkind, Eickhoff, Oathes, Jiang, Chang, Jones-Hagata and Etkin2015), chronicity (Vollebergh et al., Reference Vollebergh, Iedema, Bijl, de Graaf, Smit and Ormel2001), developmental change (McElroy, Belsky, Carragher, Fearon, & Patalay, Reference McElroy, Belsky, Carragher, Fearon and Patalay2018), functional impairment (Waszczuk, Kotov, Ruggero, Gamez, & Watson, Reference Waszczuk, Kotov, Ruggero, Gamez and Watson2017a), treatment planning (Waszczuk et al., Reference Waszczuk, Zimmerman, Ruggero, Li, MacNamara, Weinberg and Kotov2017b) and treatment response (Andrews et al., Reference Andrews, Goldberg, Krueger, Carpenter, Hyman, Sachdev and Pine2009). Initially comprising ‘Internalizing’ and ‘Externalizing’ dimensions (Krueger, Caspi, Moffitt, & Silva, Reference Krueger, Caspi, Moffitt and Silva1998), and then a ‘Thought Disorder’ (psychosis) dimension (Kotov et al., Reference Kotov, Chang, Fochtmann, Mojtabai, Carlson, Sedler and Bromet2011), more recent models have introduced ‘Detachment’ and ‘Somatoform’ dimensions, and bifurcated Internalising into ‘Fear’ and ‘Distress’ (Lahey et al., Reference Lahey, Applegate, Hakes, Zald, Hariri and Rathouz2012) sub-factors, and Externalising into ‘Disinhibited’ and ‘Antagonistic’ sub-factors (Wright & Simms, Reference Wright and Simms2015). A general dimension of psychopathology, termed ‘p’ (Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel and Moffitt2014; Lahey et al., Reference Lahey, Applegate, Hakes, Zald, Hariri and Rathouz2012), was introduced to explain the covariation between all lower-order dimensions and has been evidenced in nationally representative samples of adults (Caspi et al., Reference Caspi, Houts, Belsky, Goldman-Mellor, Harrington, Israel and Moffitt2014; Lahey et al., Reference Lahey, Applegate, Hakes, Zald, Hariri and Rathouz2012; Martel et al., Reference Martel, Pan, Hoffmann, Gadelha, do Rosario, Mari and Salum2017), children and adolescents (Carragher et al., Reference Carragher, Teesson, Sunderland, Newton, Krueger, Conrod and Slade2016; Laceulle, Vollebergh, & Ormel, Reference Laceulle, Vollebergh and Ormel2015; Lahey et al., Reference Lahey, Rathouz, Keenan, Stepp, Loeber and Hipwell2015; Martel et al., Reference Martel, Pan, Hoffmann, Gadelha, do Rosario, Mari and Salum2017; McElroy et al., Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Patalay et al., Reference Patalay, Fonagy, Deighton, Belsky, Vostanis and Wolpert2015; Tackett et al., Reference Tackett, Lahey, van Hulle, Waldman, Krueger and Rathouz2013; Waldman, Poore, van Hulle, Rathouz, & Lahey, Reference Waldman, Poore, van Hulle, Rathouz and Lahey2016) and among clinical patients (Hyland et al., Reference Hyland, Murphy, Shevlin, Carey, Vallieres, Murphy and Elklit2018a; Reininghaus, Priebe, & Bentall, Reference Reininghaus, Priebe and Bentall2013; Reininghaus et al., Reference Reininghaus, Böhnke, Hosang, Farmer, Burns, McGuffin and Bentall2016). In each of these studies, p was modelled as a bifactor dimension (see Fig. 1a).
Fig. 1. (a) Bifactor order model of psychopathology. (b) Higher-order model of psychopathology.
The dimensional approach to psychopathology achieved prominence with the publication of the Hierarchical Taxonomy of Psychopathology (HiTOP: Kotov et al., Reference Kotov, Krueger, Watson, Achenbach, Althoff, Bagby and Zimmerman2017). HiTOP proposes that ‘symptoms’ cluster together into correlated ‘syndromes’, and these syndromes are manifestations of higher-order ‘sub-factors’ (e.g. Fear and Distress). All sub-factors are subsumed under a small number of broad ‘spectra’ dimensions (e.g. Internalising, Externalising, Thought Disorder), and p sits at the top of this hierarchy capturing the covariation between the spectra-level dimensions. Considerable evidence has accumulated in support of the hierarchical structure proposed by HiTOP (Conway, Latzman, & Krueger, Reference Conway, Latzman and Krueger2019a; Forbes et al., Reference Forbes, Kotov, Ruggero, Watson, Zimmerman and Krueger2017; Kim & Eaton, Reference Kim and Eaton2015; Kotelnikova, Weaver, & Clark, Reference Kotelnikova, Weaver and Clark2019). Consequently, two approaches to modelling a general dimension of psychopathology exist, and they make different assumptions about the fundamental nature of psychopathology (van Bork, Epskamp, Rhemtulla, Borsboom, & van der Maas, Reference van Bork, Epskamp, Rhemtulla, Borsboom and van der Maas2017); one models p as a bifactor dimension, and the other models p as a higher-order dimension.
In the bifactor model, symptom variation and covariation are explained by one general dimension (p) and multiple specific dimensions (e.g. Internalising, Externalising and Thought Disorder) that are orthogonal to the general dimension (Fig. 1a). The general and specific dimensions directly affect symptoms, and thus ‘compete’ to capture symptom variation and covariation. In the higher-order model, symptom variation and covariation are explained by multiple lower-order dimensions (e.g. Internalising, Externalising and Thought Disorder) and the correlations between these dimensions are explained by one superordinate dimension (Fig. 1b). In this model, the general and specific dimensions do not ‘compete’ to capture symptom variation/covariation, nor are they orthogonal to one another. Rather, p causes variation in the lower-order dimensions and indirectly affects symptoms via these subordinate factors.
Greene et al. (Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019) showed that between 2010 and 2017, 95% of studies comparing these representations of psychopathology found that the bifactor model had superior overall model fit. However, using simulated data, they demonstrated that standard indices of model fit, and model comparison, exhibit a ‘pro-bifactor’ bias. Greene et al. (Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019) showed that when the true underlying model was a correlated factor model, standard model fit, and comparison indices consistently favoured a bifactor model. These and other results (Markon, Reference Markon2019; Reise, Kim, Mansolf, & Widaman, Reference Reise, Kim, Mansolf and Widaman2016) demonstrate that researchers should not rely solely on estimates of model fit in order to ‘pick the right model’ when comparing bifactor models to correlated factor models or higher-order models. Rather, it is necessary that models be subjected to ‘riskier’ tests of validity (Meehl, Reference Meehl1978), including how they perform in relation to external variables. Moreover, as there is evidence that bifactor models produce spurious evidence of superior model fit because of their ability to accommodate model misspecifications (Reise et al., Reference Reise, Kim, Mansolf and Widaman2016), and randomness in the data (Bonifay & Cai, Reference Bonifay and Cai2017), bifactor dimensions should be assessed in terms of their reliability and replicability (Rodriguez, Reise, & Haviland, Reference Rodriguez, Reise and Haviland2016). Existing data shows that when p is modelled as a bifactor, it is correlated with – and predictive of – an array of exogenous variables (Caspi & Moffitt, Reference Caspi and Moffitt2018). However, fewer studies have examined the reliability and replicability of the dimensions of psychopathology in a bifactor model. In those that have, there is consistent support for the reliability and replicability of p, however, less consistent support has been obtained for the specific dimensions (Constantinou et al., Reference Constantinou, Goodyer, Eisler, Butler, Kraam, Scott and Fonagy2019; Martel et al., Reference Martel, Pan, Hoffmann, Gadelha, do Rosario, Mari and Salum2017; McElroy et al., Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Murray, Eisner, & Ribeaud, Reference Murray, Eisner and Ribeaud2016; Watts, Poore, & Waldman, Reference Watts, Poore and Waldman2019).
Current study
The goal of this study was to evaluate whether p is better represented as a bifactor dimension or as a higher-order dimension of psychopathology. We followed the recommendations of Greene et al. (Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019) and Watts et al. (Reference Watts, Poore and Waldman2019) and extended our assessments beyond tests of overall model fit to also include ‘riskier’ tests of model performance. Thus, we first assessed the overall model fit of bifactor and higher-order models of psychopathology (along with a unidimensional model and multiple correlated factor models). Based on the existing literature (Caspi & Moffitt, Reference Caspi and Moffitt2018), we hypothesised that all models would provide a satisfactory representation of the data, however, given the pro-bifactor bias associated with standard model fit and comparison indices (Greene et al., Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019), we hypothesised that a bifactor model would ‘best’ fit the data. The reliability and replicability of the general and specific dimensions of psychopathology from the best-fitting bifactor model were then assessed, as per the recommendations of Rodriguez et al. (Reference Rodriguez, Reise and Haviland2016). Based on existing evidence, it was hypothesised that p would have excellent reliability and replicability, however, the specific dimensions would yield less robust results.
Second, we evaluated how the dimensions of psychopathology – p and the specific/first-order dimensions – estimated within the bifactor and higher-order models of psychopathology were correlated with one another. Based on Kim and Eaton's (Reference Kim and Eaton2015) findings, we hypothesised that (a) bifactor p and higher-order p would be nearly perfectly correlated and (b) the specific and first-order dimensions would be highly correlated when modelled within the bifactor and higher-order models, respectively.
Third, the associations between multiple external variables for mental illness and p, modelled as a bifactor dimension and as a higher-order dimension, were assessed using structural equation modelling (s.e.m.). Under the assumption that p is almost identical when modelled as a bifactor dimension or as a higher-order dimension, it was hypothesised that the observed patterns of association with the external variables would be similar within both modelling approaches. The associations between the external risk variables and the specific (bifactor model) and first-order (higher-order model) dimensions were also assessed. As there is limited evidence regarding the relative associations between specific and first-order dimensions and external risk variables, no hypotheses were formed for this part of the analyses.
Methods
Participants and procedure
The sample for this study was drawn from a panel of research participants that is representative of the general adult population of the United Kingdom (UK), as per the most recent 2011 census. A survey company, Qualtrics, was employed and quota sampling was used to gather a sample that was representative of the UK adult population in terms of age and geographical distribution (England, Wales, Scotland and Northern Ireland). These data were collected in 2017 as part of a larger project examining trauma-related psychopathology. There were three inclusion criteria: participants had experienced a traumatic life event, were born in the UK, and were 18 years of age or older. In total, 2653 panel members were contacted by Qualtrics via email and asked to participate, and 1051 consented and met the inclusion criteria. Ethical approval was granted by the ethics committee of the institution to which the first author was affiliated at the time of the data collection. The mean age was 47.18 years (s.d. = 15.00, range = 18–90 years), and 68.4% were female. Nearly half ‘grew up in an urban/large city area’ (45.3%), 70.4% were ‘in a committed relationship’, 32.5% had ‘children under 16’, 62.7% completed a college/university education, 68.5% were ‘employed’ and 17.8% had emigrated at some point in their life.
Measures
The 49 symptoms of psychopathology measured in this study are presented in online Supplementary Table S1. These were taken from self-report questionnaires that mixed binary and Likert-scale response options. While the analysis could have been conducted on the basis of polyserial correlations, this would have meant that variables were measured on different conceptual/clinical levels. In order to harmonise all variables, it was decided to make all variables binary, and employ cut-off scores that represented what would be clinically meaningful. Thus, all items were dichotomised to reflect the ‘presence’ (1) or ‘absence’ (0) of a symptom and these transformations were based on scoring guidelines and standard research procedures.
Internalising
There were 37 symptoms of internalising psychopathology. Twelve were taken from the International Trauma Questionnaire (Cloitre et al., Reference Cloitre, Shevlin, Brewin, Bisson, Roberts, Maercker and Hyland2018), a measure of ICD-11 Complex Posttraumatic Stress Disorder (CPTSD). Six items measure PTSD symptoms and six measure ‘Disturbances in Self-Organization’ symptoms. All items were answered on a five-point Likert scale (0 = ‘Not at all’ to 4 = ‘Extremely’), and symptom endorsement was based on a score of ⩾2 (‘Moderately’). Nine Major Depression and seven Generalised Anxiety Disorder symptoms were measured using the Patient Health Questionnaire-9 (Kroenke, Spitzer, & Williams, Reference Kroenke, Spitzer and Williams2001) and the Generalised Anxiety Disorder 7-Item Scale (Spitzer, Kroenke, & Williams, Reference Spitzer, Kroenke and Williams2006), respectively. Both employ a four-point Likert scale (0 = ‘Not at all’ to 3 = ‘Nearly every day’), and symptom endorsement was based on score ⩾1 (‘Several days’). Nine Borderline Personality Disorder symptoms were assessed using a self-report measure based on the BPD screening module of the Structured Clinical Interview for DSM-IV Axis II disorders (Hyland, Shevlin, Fyvie, & Karatzias, Reference Hyland, Shevlin, Fyvie and Karatzias2018b). Respondents indicated whether each symptom was ‘true’ (1) or ‘not true’ (0) of them.
Externalising
There were five indicators of externalising psychopathology. The three-item AUDIT Alcohol Consumption Questionnaire (Bush, Kivlahan, McDonell, Fihn, & Bradley, Reference Bush, Kivlahan, McDonell, Fihn and Bradley1998) was used to assess frequency of alcohol use (0 = ‘Never or less than monthly’, 1 = ‘2–3 times per month or more frequently’), daily consumption of alcohol (0 = ‘Less than 2 units per day’, 1 = ‘More than 2 units per day’) and frequency of binge drinking (0 = ‘Never’, 1 = ‘Sometimes or regularly’). Two questions measuring frequency of cannabis use (0 = ‘Never/once or twice in my life’, 1 = ‘A few times a year to every day’) and use starting before 18 (0 = ‘No’, 1 = ‘Yes’) were taken from the UK's 2007 Adult Psychiatric Morbidity Survey (Mc Manus, Meltzer, Brugha, Bebbington, & Jenkins, Reference Mc Manus, Meltzer, Brugha, Bebbington and Jenkins2009).
Thought Disorder
The Adolescent Psychotic-Like Symptom Screener (Kelleher, Harley, Murtagh, & Cannon, Reference Kelleher, Harley, Murtagh and Cannon2011) includes seven items measuring the frequency of different ‘positive’ psychosis experiences. A four-point Likert-scale (0 = ‘Never’ to 3 = ‘Nearly always’) was used and scores ⩾1 (‘Sometimes’) indicated the presence of a psychotic-like symptom.
Traumatic exposure
The Adverse Childhood Experiences (ACE) questionnaire (Felitti et al., Reference Felitti, Anda, Nordenberg, Williamson, Spitz, Edwards and Marks1998) and the Life Events Checklist for DSM-5 (LEC-5; Weathers et al., Reference Weathers, Litz, Keane, Palmieri, Marx and Schnurr2013) were used to measure childhood (‘before 18’) and adulthood (‘18 or older’) interpersonal and non-interpersonal trauma, respectively. Five ACE (verbal abuse, physical abuse, sexual abuse, emotional neglect and physical neglect; all by a caregiver) and six LEC-5 (physical assault, assault with a weapon, sexual assault, other sexual experiences, captivity and causing serious injury/death to another person) events were used to measure childhood interpersonal trauma. Adulthood interpersonal trauma was measured using the same six LEC-5 events. Five ACE (parental divorce, witnessing domestic violence, family member with a drug/alcohol problem, family member with a serious mental illness and a family member in prison) and seven LEC-5 (natural disaster, fire/explosion, transportation accident, serious accident, exposure to toxic substance, life-threatening illness/injury and sudden/unexpected death of a loved one) events were used to measure childhood non-interpersonal trauma. Adulthood non-interpersonal trauma was measured using the same seven LEC-5 events.
Data analysis
The latent structure of the 49 symptoms of psychopathology was assessed using confirmatory factor analysis (CFA) and confirmatory bifactor modelling. Six models were tested: (1) a unidimensional model with one general factor (p); (2) a correlated three-factor model including dimensions of ‘Internalizing’, ‘Externalizing’ and ‘Thought Disorder’; (3) a correlated four-factor model including dimensions of ‘Fear’ (PTSD and anxiety symptoms), ‘Distress’ (depression, disturbances of self-organisation and borderline personality symptoms), ‘Externalizing’ and ‘Thought Disorder’; (4) a higher-order model with four first-order factors (‘Fear’, ‘Distress’, ‘Externalizing’ and ‘Thought Disorder’) and one second-order factor (p); (5) a bifactor model with one general factor (p) orthogonal to three correlated specific factors (‘Internalizing’, ‘Externalizing’ and ‘Thought Disorder’) and (6) a bifactor model with one general factor (p) orthogonal to four correlated specific factors (‘Fear’, ‘Distress’, ‘Externalizing’ and ‘Thought Disorder’). It was not possible to test a higher-order model with three first-order factors as this is statistically indistinguishable from a correlated first-order model.
Following the identification of the best-fitting bifactor model, the reliability and replicability of each dimension were assessed in accordance with the recommendations of Rodriguez et al. (Reference Rodriguez, Reise and Haviland2016). Omega reliability (ω; proportion of common variance explained by the general and specific factors), omega hierarchical [ωH; proportion of variance within the symptom indicators attributable to the general (or specific) factor(s), controlling for the specific (or general) factors], relative omega (ωR: represents the proportion of reliable variance due to the general factor independent of the specific factors, and each specific factor independent of the general factor) and index H (the extent to which a set of items represents a latent variable and the likelihood of that latent variable replicating across studies) were calculated. Omega coefficients and index H values range from 0–1, and values ⩾0.80 reflect satisfactory reliability and replicability (Rodriguez et al., Reference Rodriguez, Reise and Haviland2016). These indices were calculated using Dueber's (Reference Dueber2017) software.
Factor scores were saved for each dimension in the best fitting bifactor model and the higher-order model, and the correlations between these factor scores were assessed.
s.e.m. was used to determine the multivariate association between the dimensions within the bifactor and higher-order models and nine exogenous risk variables: sex (0 = male, 1 = female), age, urbanicity (0 = grew up in a rural area, 1 = grew up in an urban area), employment status (0 = employed/retired/student/homemaker, 1 = unemployed), number of different childhood interpersonal traumas, childhood non-interpersonal traumas, adulthood interpersonal traumas and adulthood non-interpersonal traumas. These risk variables were treated as observed variables and the latent factors of psychopathology were regressed onto each simultaneously.
Analyses were performed in Mplus 8.2 (Muthén & Muthén, Reference Muthén and Muthén2017) using the Weighted Least Squares Mean- and Variance-Adjusted estimator which is appropriate for categorical level indicators (Flora & Curran, Reference Flora and Curran2004). There was minimal missing data (0.19%) and it was handled using pairwise deletion. Model fit was evaluated by several standard goodness-of-fit indices (Hu & Bentler, Reference Hu and Bentler1999): a non-significant chi-square (χ2) result indicates excellent model fit; comparative fit index (CFI) and Tucker–Lewis index (TLI) values ⩾0.90 and root mean square error of approximation (RMSEA) and standardised root mean square residual (SRMR) values ⩽0.08, indicate acceptable mode fit, respectively. All models were re-estimated each model using the maximum likelihood estimator to generate Bayesian information criterion (BIC) values which can be used to compare nested and non-nested models. The model with the lowest BIC value is considered to be statistically superior, and a difference of 10 points is considered evidence for the superiority of the model with the lower value (Raftery, Reference Raftery1995). However, simulation studies have shown that the BIC favours bifactor models even when the true underlying model is not a bifactor (Greene et al., Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019).
Results
Descriptive statistics
Symptom endorsement rates are presented in online Supplementary Table S1, and these ranged from 8.1% (‘Have you ever had messages sent just to you through the TV or radio?’) to 71.4% (‘Feeling tired or having little energy’). The distributions of childhood interpersonal traumas (M = 2.14, s.d. = 2.38), childhood non-interpersonal traumas (M = 1.72, s.d. = 1.86), adulthood interpersonal traumas (M = 0.78, s.d. = 1.18) and adulthood non-interpersonal traumas (M = 1.31, s.d. = 1.28) were positively skewed.
The latent structure of psychopathology
Model fit results are presented in Table 1. The unidimensional model (Model 1) had acceptable fit according to the CFI, TLI and RMSEA results, however, the SRMR result indicated poor fit. The three- and four-factor correlated models (Models 2 and 3) had acceptable fit across all indices, and both were superior to the unidimensional model. The BIC was lower for the four-factor model than the three-factor model, indicating improvement in fit when the internalising dimension was divided between Fear and Distress. The higher-order model of p (Model 4) also had acceptable fit, however, the BIC was higher than for the correlated four-factor model. The bifactor models had superior fit to all other models, and the BIC was lowest for the model with one general factor and four specific factors (Model 6).
Table 1. Model fit results for the alternative dimensional models of the structure of psychopathology
I, Internalising; D, Distress; F, Fear; E, Externalising; TD, Thought Disorder; p, General Psychopathology; χ2, chi square goodness of fit statistic; df, degrees of freedom; CFI, Comparative Fit Index; TLI, Tucker Lewis Index; RMSEA (90% CI), Root-Mean-Square Error of Approximation with 90% confidence intervals; SRMR, Standardised Root Mean Square Residual; BIC, Bayesian Information Criterion.
Note. N = 1049.
*Indicates χ2 test is statistically significant (p < 0.001).
The model parameters for the bifactor and higher-order models are presented in Table 2. In the bifactor model, all symptoms bar one loaded positively and significantly onto p. Eight of the 13 fear symptoms loaded onto the Fear dimension; 21 of the 24 distress symptoms loaded onto the Distress dimension; four of the five externalising symptoms loaded onto the Externalising dimension and all of the psychosis symptoms loaded onto the Thought Disorder dimension. In general, the majority of the Fear and Distress symptoms loaded more strongly onto p than onto their respective specific dimension, while the opposite was the case for the Externalising and Thought Disorder symptoms. The correlations between the specific dimensions were all weak.
Table 2. Standardised factor loadings and factor correlations for the bifactor and higher-order models of p
Note: All factor loadings and factor correlations are statistically significant (p < 0.05) except for those marked ns.
In the higher-order model, every symptom loaded positively and significantly onto its respective first-order factor, and the four first-order factors loaded positively and significantly onto the second order p factor.
Table 3 includes the reliability and replicability estimates for the bifactor dimensions. Each dimension had satisfactory – or near satisfactory – construct replicability (i.e. index H values ⩾0.80), however, only p exhibited satisfactory reliability (i.e. ω values ⩾0.80). Fear and Distress, in particular, had extremely low levels of reliability indicating that these dimensions accounted for very little reliable item variance, adjusting for p.
Table 3. Reliability and construct replicability results for the bifactor dimensions of psychopathology
Note: ω = omega reliability; ωH = omega hierarchical reliability; ωR = relative omega reliability; H = construct replicability.
Associations between bifactor and higher-order dimensions
The correlations between the factor scores of each dimension in the bifactor and higher-order models are presented in Table 4. The correlation between the bifactor and higher-order representations of p was almost perfect. Similarly, bifactor p was nearly perfectly correlated with the higher-order Fear and Distress dimensions. Contrastingly, higher-order p was very weakly associated with the specific factors in bifactor model. The Externalising and Thought Disorder dimensions were strongly associated across the two models, whereas, the Fear and Distress dimensions were very weakly associated across the two models.
Table 4. Correlations between the dimensions from the bifactor and higher-order models of psychopathology
Note: all correlations are statistically significant (p < 0.05) except for those marked ns.
Associations with exogenous variables
The multivariate associations between the nine external risk variables and the dimensions of psychopathology in the bifactor and higher-order models of psychopathology are presented in Table 5. The s.e.m. model based on the bifactor representation of p provided an acceptable fit of the data [χ2 (1468) = 3001, p < 0.001; CFI = 0.971; TLI = 0.969; RMSEA = 0.032 (90% CI 0.030–0.033), SRMR = 0.061; BIC = 41 069], as did the second-order model of p [χ2 (1555) = 4920, p < 0.001; CFI = 0.937; TLI = 0.935; RMSEA = 0.045 (90% CI 0.044–0.047), SRMR = 0.092; BIC = 42 839]. To generate the associations between the external risk variables and the four first-order dimensions, the s.e.m. model was rerun without the second-order p factor, and this model also provide a satisfactory representation of the data [χ2 (1526) = 4863, p < 0.001; CFI = 0.938; TLI = 0.934; RMSEA = 0.046 (90% CI 0.044–0.047), SRMR = 0.085; BIC = 42 769].
Table 5. Standardised regression coefficients between each risk variable and each dimension of psychopathology (N = 1049)
p, general psychopathology; F, Fear; D, Distress; E, Externalising; TD, Thought Disorder.
a Standardised regression coefficients for p were derived from the s.e.m. model based on the second-order model, and the effects for F, D, E and TD were derived from the s.e.m. model based on the first-order model.
Statistical significance = *p < 0.05, **p < 0.01, ***p < 0.001.
R 2 = the percentage of variance in each dimension explained by the nine risk variables.
The nine risk variables explained 30.8% of the variance in p when modelled as a bifactor dimension, and 40.7% of variance in p when modelled as a higher-order dimension. Furthermore, the nine risk variables explained substantially more variance in the Fear, Distress and Thought Disorder dimensions in the correlated-factors model compared to these specific dimensions in the bifactor model. The risk variables explained a similar proportion of variance in the Externalising dimension within the bifactor and correlated-factor models.
The strength and direction of the associations between the external risk variables and bifactor p and higher-order p were similar. In general, however, the associations were marginally stronger when p was modelled as a higher-order dimension. In both cases, p was most strongly associated with childhood interpersonal traumas (bifactor p, β = 0.27, and, higher-order p, β = 0.32). Bifactor and higher-order p were also positively correlated with all other forms of trauma and younger age.
Some notable differences emerged between the specific dimensions in the bifactor model and the first-order dimensions in the correlated model, especially for the Fear and Distress dimensions. In the bifactor model, Fear was associated with growing up in an urban area, and higher levels of childhood and adulthood interpersonal trauma. In the correlated model, Fear was associated with all forms of trauma, younger age and female sex. In the bifactor model, Distress was associated with male sex, younger age, higher levels of childhood and adulthood interpersonal trauma and lower levels of adulthood non-interpersonal trauma. In the correlated model, however, Distress was associated with younger age, higher levels of childhood and adulthood interpersonal trauma and higher levels of childhood non-interpersonal trauma. The associations for the Externalising and Thought Disorder dimensions were similar across the two models although some difference did emerge. Externalising was associated with younger age in the correlated model, but not the in the bifactor model; and Thought Disorder was correlated with childhood interpersonal trauma and childhood non-interpersonal trauma in the correlated model, but not in the bifactor model.
Discussion
The existing literature describes two ways in which a general dimension of psychopathology (p) may be incorporated into a comprehensive account of the latent structure of psychopathology. Although the majority of studies have found that the bifactor approach to modelling p provides superior model fit to the hierarchical approach favoured by the HiTOP theory, there is compelling evidence to be sceptical of these findings (Greene et al., Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019). As such, the primary objective of this study was to compare how the bifactor and hierarchical approaches to modelling p performed when assessed in relation to a set of well-established risk variables for mental illness. Whether p should be modelled as a higher-order dimension or as a bifactor dimension might appear to be a niche statistical question with little relevance to clinical reality, however, this is not the case. These models refer to the same constructs (p, Internalising, Externalising, Thought Disorder) and this can provide the illusion that the constructs contained therein reflect the same underlying psychological phenomena, but the reality is quite different (see van Bork et al., Reference van Bork, Epskamp, Rhemtulla, Borsboom and van der Maas2017 for full discussion). In the bifactor model, the general dimension of psychopathology is orthogonal to the specific dimensions whereas in the higher-order model the general dimension is causally related to the specific dimensions. This means that clinicians and researchers would have to think in very different ways about what these constructs reflect, how they relate to one another, how best to assess them, how to approach treatment interventions and how exogenous risk variables affect each dimension, depending upon which model is more accurate. Thus, determining the correct approach to modelling p is a matter of utmost importance for clinicians, researchers and patients.
We modelled the latent structure of 49 symptoms of psychopathology from a trauma-exposed community sample, and consistent with the existing evidence (Caspi & Moffitt, Reference Caspi and Moffitt2018; Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019b), the multidimensional, higher-order and bifactor models yielded satisfactory model fit results. Our findings showed that there was value in bifurcating the Internalising dimension between its Fear and Distress components, similar to previous findings (e.g. Lahey et al., Reference Lahey, Applegate, Hakes, Zald, Hariri and Rathouz2012). However, such findings are likely to be dependent upon the specific indicators available in any given study. Presumably, had we a larger set of indicators of Externalising psychopathology, we may have found evidence to make a distinction between its Disinhibited and Antagonistic sub-factors. Regardless, the addition of a higher-order p factor resulted in a plausible representation of the data, and all first-order factors loaded significantly onto p. Higher-order p captured a substantial proportion of variance in each of the first-order factors, with the exception of Externalising. As predicted, the bifactor models provided the closest fit to the data, and one notable finding from Greene et al.'s (Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019) simulation work is worth highlighting here. They showed that pro-bifactor bias in all fit indices was common under conditions of unmodelled cross-factor loadings. Inspection of the modification indices for the correlated and higher-order models showed numerous instances of very strong unmodelled cross-factor loadings. The superior fit for the bifactor models in this study is, therefore, consistent with Greene et al.'s (Reference Greene, Eaton, Li, Forbes, Krueger, Markon and Fried2019) (p. 756) conclusion that ‘the mistaken inference of bifactor superiority seems to be driven by the general dimension's erroneous accommodation of misspecifications through capturing theoretically unexplained variance and repackaging it as common variance, even though it is not.’
The reliability and replicability analyses provided additional evidence to be cautious of the favourable model fit results for the bifactor model. The general dimension accounted for 92% of reliable variance among the symptoms of psychopathology, independent of the variance accounted for by the specific dimensions. These findings add to similar observations from child, adolescent and adult samples (Constantinou et al., 2019; Martel et al. Reference Martel, Pan, Hoffmann, Gadelha, do Rosario, Mari and Salum2017; McElroy et al. Reference McElroy, Belsky, Carragher, Fearon and Patalay2018; Murray et al., Reference Murray, Eisner and Ribeaud2016; Watts et al. Reference Watts, Poore and Waldman2019). However, when the variance attributable to the general dimension was partitioned out, the Fear (11%) and Distress (14%) dimensions explained little reliable variances in their respective symptoms, while Externalising (70%) and Thought Disorder (66%) explained a higher, but less than satisfactory, level of reliable variance in their symptoms. These findings suggest that the vast majority of the Internalising-based symptoms are saturated by p and call into question the conceptual integrity of the Fear and Distress dimensions. Given that Fear and Distress reflect little of the variance in their respective symptom indicators, one may reasonably wonder if these dimensions are truly reflective of Fear and Distress based psychopathology.
This concern was heightened by the correlations observed between the bifactor dimensions and their counterparts from the higher-order model. The two Fear dimensions shared just 3.6% of variance, and the two Distress dimensions shared 15.2% of variance. It is difficult to see how these dimensions can be considered equivalent despite the same names being used to describe the constructs. On the other hand, the general dimension of psychopathology in the bifactor and higher-order models shared 94.1% of variance; a result consistent with Kim and Eaton's (Reference Kim and Eaton2015). Thus, current and past findings indicate that whether p is modelled as directly affecting psychopathology symptoms (as in the bifactor model) or indirectly affecting these symptoms via subordinate dimensions (as in the higher-order model), the two parameterisation methods produce near equivalent results.
The s.e.m. findings indicated that p may operate in a slightly more advantageous manner when modelled within a hierarchical framework. The mental health risk variables explained 10% more variance in higher-order p compared to bifactor p. Additionally, while both parameterisations of p produced consistent correlations with the external variables, the strength of some of these associations – notably with age and interpersonal traumas – were slightly stronger for higher-order p. Additionally, all associations between the risk variables and the first-order dimensions of psychopathology in the higher-order model were easily interpretable, and consistent with the wider mental health literature; whereas, some odd and counterintuitive findings emerged for the specific factors in the bifactor model. For example, higher levels of Distress were associated with lower levels of adulthood non-interpersonal traumas, and Thought Disorder was not associated with childhood interpersonal, or non-interpersonal, trauma. The latter is a particularly perplexing result given the extensive literature demonstrating that childhood trauma is strongly – and probably causally – related to psychotic illness (Varese et al., Reference Varese, Smeets, Drukker, Lieverse, Lataster, Viechtbauer and Bentall2012). These results not only provide support for the higher-order model over the bifactor model of psychopathology in terms of explaining psychopathology risk, but they also highlight how trauma exposure – in childhood and in adulthood – is a critical risk-factor for transdiagnostic and transdimensional psychopathology.
Focusing on the higher-order model, some interesting associations emerged between the sociodemographic variables and the dimensions of psychopathology. Younger age was associated with all of the first-order dimensions, and p. There was no sex difference on p, but women had higher levels of Fear and men had higher levels of Externalising and Thought Disorder. Growing up in an urban area was only associated with Thought Disorder, consistent with previous research about the importance of exposure to urban environments in psychosis (Vassos, Pedersen, Murray, Collier, & Lewis, Reference Vassos, Pedersen, Murray, Collier and Lewis2012). Continuing to identify which aspects of the urban environment impact on the Thought Disorder dimension is important given that human beings are becoming an increasingly urban species (United Nations, 2019). Unemployment status was only associated with the Externalising dimension, and may suggest that Externalising psychopathology brings about higher levels of impairment than other forms of psychopathology. Additional research is required to quantify the degree of impairment associated with the different dimensions of psychopathology.
Several limitations should be noted. First, our data were derived from trauma-exposed members of the general population therefore they do not generalise to the entire population. Second, we were only able to use a limited number of symptom indicators for the Externalising and Thought Disorder dimensions, and had no items to model other dimensions such as ‘Somatoform’ and ‘Detachment’. Having additional measures to represent these dimensions would have better approximated the full HiTOP model. The development of a comprehensive method of measuring all aspects of the HiTOP model remains an important objective (Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019b). Third, we relied on questionnaire guidelines and standard research practices to dichotomise symptoms as being ‘present’ or ‘absent’, thus the endorsement rates are likely biased due to measurement error. Replication of these results using ordinal and continuous indicators of psychopathology symptoms will be important.
Conclusion
Categorical models of psychopathology have dominated the empirical and clinical landscape for the last century, and as their limitations have become increasingly well recognised, dimensional models of psychopathology offer promise in more accurately describing the fundamental nature of psychopathology. As such, dimensional models of psychopathology may lead to important advances regarding the causes and consequences of mental illness, and how best to prevent and treat mental illness (Conway et al., Reference Conway, Forbes, Forbush, Fried, Hallquist, Kotov and Eaton2019b; Ruggero et al., Reference Ruggero, Kotov, Hopwood, First, Clark, Skodol and Zimmermann2019). The empirical literature supports the existence of a general dimension of psychopathology that captures variance and covariance shared across all forms of mental illness, however, alternative approaches to incorporating p into a dimensional theory of psychopathology have been proposed. These alternative approaches have important implications for clinical theory and practice, and it is essential that psychological science determines the most appropriate way to incorporate p within a theory of psychopathology. Our findings indicate that the hierarchical approach outlined by the HiTOP theory is the better approach to modelling p. More work is needed to determine the psychological mechanisms that underlie p but Craver, Johnson, and Timpano (Reference Craver, Johnson and Timpano2018) suggest that it may be explained within a dual process framework as over-reactivity of associative processes to emotion-triggering events; a hypothesis that might help to explain why p is so strongly associated with childhood interpersonal trauma. The specification of mechanisms involved in p might aid clinicians in identifying (a) who is most at risk for different forms of psychopathology and (b) how best to intervene to mitigate different mental health problems.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S003329172000104X.
