Introduction
Current ultra-high-risk (UHR) criteria, emphasizing recent onset or worsening of subthreshold psychotic symptoms (Miller et al. Reference Miller, McGlashan, Rosen, Somjee, Markovich, Stein and Woods2002, Reference Miller, Zipursky, Perkins, Addington, Woods, Hawkins, Hoffman, Preda, Epstein, Addington, Lindborg, Marquez, Tohen, Breier and McGlashan2003; Yung et al. Reference Yung, Yuen, McGorry, Phillips, Kelly, Dell'Olio, Francey, Cosgrave, Killackey, Stanford, Godfrey and Buckby2005, Reference Yung, Nelson, Stanford, Simmons, Cosgrave, Killackey, Phillips, Bechdolf, Buckby and McGorry2008), appear to be insufficient in predicting imminent onset of first-episode psychosis, as about 20% show a psychotic outcome at 2 years follow-up (Fusar-Poli et al. Reference Fusar-Poli, Bonoldi, Yung, Borgwardt, Kempton, Valmaggia, Barale, Caverzasi and McGuire2012). Therefore, the challenge is to develop a prognostic model to more precisely identify those individuals most likely to make a transition from UHR to a first-episode psychosis. In this context it should also be noted that the UHR stage could be divided into substages, in which people have progressed to an even more elevated stage of an imminent risk of a first psychosis. In each of these stages a different set of prognostically relevant factors could play a role. In the present study we aimed to increase predictive accuracy, while also taking into account the specific UHR stage. This stage-dependent prognostic modelling approach is relevant, because it can help improve the continuity of well-tailored (personalized) proactive care for individuals with a poor prognosis.
The PACE400 Study (Nelson et al. Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner, Broussard, Simmons, Foley, Brewer, Francey, Amminger, Thompson, McGorry and Yung2013) and two large multi-center studies aimed to identify specific predictors of transition to psychosis. The PACE400 Study included a cohort of UHR patients recruited to participate in research studies between 1993 and 2006. A two-factor prediction rule was obtained: poor functioning at baseline, and the duration of symptoms before clinic entry. Patients at UHR with either one or both of these factors had a chance of 39%, 52%, or 72% of developing psychosis within, respectively, 1, 3 or 5 years. However, similar to other models attempting to narrow the criteria (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter and Klosterkötter2010a ), the loss of sensitivity was unfavourably high (McGorry et al. Reference McGorry, Yung, Bechdolf and Amminger2008).
The North American Prodrome Longitudinal Study (NAPLS; Cannon et al. Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker, Seidman, Perkins, Tsuang, McGlashan and Heinssen2008), found five baseline variables contributing uniquely to the prediction of psychosis: a genetic risk for schizophrenia with recent functional decline, higher levels of unusual thought content, higher levels of paranoia, greater social impairment, and history of any drug abuse. In their population, algorithms combining two or three of these variables maximized the positive predictive value (PPV) to 74–81% compared with 35% for the UHR criteria alone. However, similar to other models (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter and Klosterkötter2010a ) and the PACE400 Study (Nelson et al. Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner, Broussard, Simmons, Foley, Brewer, Francey, Amminger, Thompson, McGorry and Yung2013), the loss of sensitivity was also high (McGorry et al. Reference McGorry, Yung, Bechdolf and Amminger2008).
The European Prediction of Psychosis Study (EPOS; Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ), developed a six-variable prediction model including the following baseline features: overall severity of positive symptoms, bizarre thinking, sleep disturbance, schizotypal personality disorder, loss of functioning in the past year, and years of education. Although the model produced an excellent positive likelihood ratio (+LR) of 19.9, again, there was considerable loss of sensitivity (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ). Using the Cox equation to calculate a prognostic score for each participant counteracted this disadvantage. Furthermore, this enabled us to introduce a prognostic index by stratifying the score and, thus, the general risk of the sample, into different risk classes, allowing both more individualized risk estimation and preservation of the sensitivity of the inclusion criteria.
However, in the above-mentioned studies, the UHR criteria used differed slightly. The PACE400 study used the Comprehensive Assessment of At Risk Mental State (CAARMS) and the sample was recruited before the introduction of the social decline criterion (Yung et al. Reference Yung, Nelson, Stanford, Simmons, Cosgrave, Killackey, Phillips, Bechdolf, Buckby and McGorry2008). The NAPLS used the Structured Interview for Prodromal Syndromes (SIPS), also without a social decline criterion. The EPOS used the SIPS and basic symptom-based criterion for cognitive disturbances (COGDIS). All three studies recruited both cases and persons not yet classified as a case. To restrict the UHR group to a sample at imminent risk of transitioning to a first episode of psychosis would be more relevant from a clinical point of view. Such a restriction would serve to recruit patients with stage 1b as described by McGorry et al. (Reference McGorry, Hickie, Yung, Pantelis and Jackson2006) in their staging model for psychotic disorders. This latter group is characterized by psychotic-like experiences with distress, help-seeking for co-morbid disorders, and reduced functioning. This is the stage where ‘caseness’ demands therapy for persons at imminent risk of psychosis (McGorry et al. Reference McGorry, Hickie, Yung, Pantelis and Jackson2006; Yung et al. Reference Yung, Stanford, Cosgrave, Killackey, Phillips, Nelson and McGorry2006).
Therefore, in the present study we aimed to develop a stage 1b-dependent prognostic model by recruiting a sample of young people aged ⩽35 years who are seeking help in mental health services for an Axis-1 or Axis-2 DSM-IV disorder and who also fulfil the criteria of the CAARMS, including low functioning.
Method
Design and recruitment
The EDIE-NL study is a randomized controlled trial (RCT) in which add-on cognitive behaviour therapy (CBT) was compared with treatment as usual (TAU) alone (Rietdijk et al. Reference Rietdijk, Dragt, Klaassen, Ising, Nieman, Wunderink, Delespaul, Cuijpers, Linszen and van der Gaag2010). Participants were recruited by screening or referral at four treatment centers between February 2008 and February 2010. Inclusion criteria of the EDIE-NL trial were composed of the UHR criteria, assessed by the CAARMS (Yung et al. Reference Yung, Yuen, McGorry, Phillips, Kelly, Dell'Olio, Francey, Cosgrave, Killackey, Stanford, Godfrey and Buckby2005, Reference Yung, Nelson, Stanford, Simmons, Cosgrave, Killackey, Phillips, Bechdolf, Buckby and McGorry2008): ‘Familial Risk’, Attenuated Psychotic Symptoms, Brief Limited Intermittent Psychotic Symptoms (BLIPS), each accompanied by a score of ⩽50 on the Social and Occupational Functioning Assessment Scale (SOFAS; Goldman et al. Reference Goldman, Skodol and Lave1992), and/or a reduction of 30% on this scale for at least 1 month in the past year. Exclusion criteria were: a psychotic episode lasting ⩾1 week, i.e. fulfilling the DSM-IV criteria of a brief psychotic episode within a time period of ⩾7 days assessed with the Schedules for Clinical Assessment in Neuropsychiatry (SCAN 2.1) (Rijnders et al. Reference Rijnders, van den Berg, Hodiamont, Nienhuis, Furer, Mulder and Giel2000); current/previous use of antipsychotic medication with a cumulative dose of ⩾15 mg haloperidol equivalents; severe learning impairment, and/or insufficient competence in the Dutch language. Details of inclusion and exclusion criteria are described elsewhere (Rietdijk et al. Reference Rietdijk, Dragt, Klaassen, Ising, Nieman, Wunderink, Delespaul, Cuijpers, Linszen and van der Gaag2010; van der Gaag et al. Reference van der Gaag, Nieman, Rietdijk, Dragt, Ising, Klaassen, Koeter, Cuijpers, Wunderink and Linszen2012).
After providing informed consent, 201 individuals agreed to participate. Five participants were removed from the analyses because, retrospectively, two turned out to be already psychotic at inclusion and three disclosed a history of psychosis during the trial. Of the remaining 196 patients enrolled in the study, 185 (88 CBT v. 97 TAU) had at least one follow-up clinical evaluation and comprised the final sample. The study was approved by the local Medical Ethics Committee for mental health service research and registered at Current Controlled Trials (ISRCTN21353122).
Interventions
Both arms of the study were treated with TAU provided for non-psychotic DSM-disorders for which they were seeking treatment. TAU was given according to the evidence-based clinical guidelines for non-psychotic Axis-1 or Axis-2 disorders. The experimental group received TAU plus individual CBT aimed at preventing the onset of a first psychosis. The intervention consisted of CBT enriched with education on dopamine supersensitivity, the effects of dopaminergic supersensitivity on perception and reasoning, and a cognitive bias awareness training (van der Gaag et al. Reference van der Gaag, Nieman and van den Berg2013).
Clinical assessments and follow-up
The present analyses used data assessed by the CAARMS (Yung et al. Reference Yung, Yuen, McGorry, Phillips, Kelly, Dell'Olio, Francey, Cosgrave, Killackey, Stanford, Godfrey and Buckby2005, Reference Yung, Nelson, Stanford, Simmons, Cosgrave, Killackey, Phillips, Bechdolf, Buckby and McGorry2008), including the SOFAS (Goldman et al. Reference Goldman, Skodol and Lave1992), the Beck Depression Inventory – II (BDI-II; Beck et al. Reference Beck, Steer and Brown1996), the Calgary Depression Scale (CDS; Guillem et al. Reference Guillem, Pampoulova, Stip, Lalonde and Todorov2005), the Personal Beliefs about Illness Questionnaire (PBIQ-R; Birchwood et al. Reference Birchwood, Jackson, Brunet, Holden and Barton2012), the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, Reference Mattick and Clarke1998), the Manchester Short Assessment of Quality of Life (MANSA; Priebe et al. Reference Priebe, Huxley, Knight and Evans1999), the Composite International Diagnostic Interview [CIDI; alcohol and drug section) (Andrews & Peters, Reference Andrews and Peters1998], and socio-demographic data including years of education and ethnicity (Rietdijk et al. Reference Rietdijk, Dragt, Klaassen, Ising, Nieman, Wunderink, Delespaul, Cuijpers, Linszen and van der Gaag2010). The seven CAARMS subscales (Positive, Negative, Cognitive Symptoms, Emotional Disturbances, Behavioural Change, Motor/Physical Change, and General Psychopathology) are rated on a 7-point intensity and frequency scale (0–6). The four positive symptoms each include a distress score on a 0–100 scale (Supplementary Table S1). The developer of the CAARMS criteria (A. Yung) extensively trained the investigators. Pairwise inter-rater concordance for CAARMS was 81%. Follow-up CAARMS assessments took place at 2, 4, 6, 9, 12, 15 and 18 months post-baseline. Transition to psychosis was operationalized as a continuation of full-blown psychotic symptoms for ⩾7 days according to the CAARMS. The Dutch version of the SCAN (Rijnders et al. Reference Rijnders, van den Berg, Hodiamont, Nienhuis, Furer, Mulder and Giel2000) was used to assess past/current psychotic disorders with a view to the exclusion criteria, as well as to establish a psychotic diagnosis in case of transition.
Statistical analysis
Statistical analyses were performed using SPSS version 21.0 (IBM Corp., USA). Comparisons of general characteristics were analysed with the Mann–Whitney U test, t tests and χ2 tests.
To develop the prediction model, Cox proportional hazard analyses were used that estimate the effect of covariates on time to transition. In line with the analyses described by Ruhrmann et al. (Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ), Nieman et al. (Reference Nieman, Ruhrmann, Dragt, Soen, van Tricht, Koelman, Bour, Velthorst, Becker, Weiser, Linszen and de Haan2014) and Cornblatt et al. (Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen, John and Correll2015), potential model predictors were selected in a sequential procedure (Hosmer et al. Reference Hosmer, Lemeshow and May1999). First, univariate Cox regression analyses were performed and predictors that were significant at a liberal level (p < 0.20) were analysed further. Second, backward multivariate Cox regression analyses were performed within each domain (p < 0.15). Third, retained covariates were entered into domain-specific multivariate backward Cox regressions (p < 0.05). Fourth, all variables with a significant test result in the previous univariate or multivariate analysis were added one-by-one to the preliminary model and kept in the model if they remained significant. Finally, the remaining covariates from steps 3 and 4 were analysed together forward and backward to exclude effects of blocking (p < 0.05), restricting the maximum number of predictors entering the final model to a 1:5 ratio of number of predictors to events, i.e. to six predictors (Vittinghoff & McCulloch, Reference Vittinghoff and McCulloch2007). Wald χ2 was used to test the significance of individual variables in the model. Recruitment strategy (referral, screening or mixed) was entered as a strata variable in all analyses. According to the state-of-the-art manual of Hosmer and Lemeshow (Hosmer et al. Reference Hosmer, Lemeshow and May1999), the CBT intervention was introduced as a possible confounding factor. Developing the model on the TAU sample and testing it on the CBT sample was not possible due to a lack of sufficient power. To circumvent the risk of an overfitted model, and to check the internal validity of the final prediction model, a bootstrap method was employed (Loughin, Reference Loughin1998; Tropsha et al. Reference Tropsha, Gramatica and Gombar2003). The approach for risk stratification is described elsewhere (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ; Nieman et al. Reference Nieman, Ruhrmann, Dragt, Soen, van Tricht, Koelman, Bour, Velthorst, Becker, Weiser, Linszen and de Haan2014). Following this approach, the resulting Cox regression equation was used to calculate individual prognostic scores. A prognostic index (PI) was generated to differentiate risk classes to aid in predicting the prognosis of UHR patients (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ). PIs were calculated for the total sample. The obtained PI was applied to the CBT and TAU subsample to analyse how the PI ranges work with or without CBT treatment. Subsequently, a log-rank test was performed to compare the risk classes on time to transition and hazard ratio (HR) (p < 0.05). Finally, the individual prognostic scores were entered into a binary logistic regression analysis to calculate sensitivity, specificity, +LR and −LR, and the PPVs and negative predictive values (NPVs) of the Cox model and model discrimination was assessed with the area under the curve (AUC).
Results
Baseline data and tests of attrition bias
Of the 185 subjects, 18 (9.7%) were lost to the 18-month follow-up. Patients with known outcome showed no significant difference (p < 0.05) compared with those lost to follow-up in terms of socio-demographic characteristics and clinical characteristics (Tables 1 and 2).
Table 1. Socio-demographic sample characteristics with known and unknown CAARMS outcome of the Dutch EDIE-NL study
CAARMS, Comprehensive Assessment of At-Risk Mental States; EDIE, Early Detection and Intervention Evaluation; s.d., standard deviation.
a All persons making a transition to psychosis during the 18-month follow-up or completing it.
b Persons with an observation time ⩽18 months and unknown CAARMS outcomes.
c Continuous variables were compared using the Mann–Whitney U test; categorical variables were compared using the χ 2 or Fisher's exact tests.
d To keep all β coefficients positive, years of education have been inverted by rank transformation in a scale from 1–17 (higher number indicates fewer years of education, e.g. 17 = 8 years of education, 16 = 9 years of education).
e Average number of addresses within a 1-km radius as a (Dutch) measure of the degree of urbanization (http://www.cbs.nl): ⩾2500 addresses/km2 = extremely urbanized.
f We used the classification of ethnicity as defined by the Dutch Bureau of Statistics (http://www.cbs.nl).
Table 2. Clinical sample characteristics in patients with known and unknown CAARMS outcome
APS, Attenuated psychotic symptoms; BDI, Beck Depression Inventory; BLIPS, brief limited and intermittent psychotic symptoms; CAARMS, Comprehensive Assessment of At-Risk Mental States; CBT, cognitive behavior therapy; CDS, Calgary Depression Scales; CIDI, Composite International Diagnostic Interview; MANSA, Manchester Short assessment of Quality of Life; PBIQ-R, Personal Beliefs about Illness Questionnaire – Revised; SIAS, Social Interaction Anxiety Scale; SOFAS, Social and Occupational Functioning Assessment Scale; TAU, treatment as usual; UHR, ultra-high risk.
a All persons transitioning to psychosis during the 18-month follow-up or completing it.
b Persons with an observation time ⩽18 months and unknown outcomes.
c Continuous variables were compared using the t test; categorical variables were compared using the χ2 or Fisher's exact test.
d Significant at a Bonferroni-adjusted α level <0.0016.
e The numbers within parentheses indicate the score range. For example, the positive symptoms scale is comprised of four subscales (unusual thought content, non-bizarre ideas, perceptual abnormalities, and disorganized speech), each with a global rating scale ranging from 0 to 6. Therefore, on the positive symptoms scale the minimum score that can be obtained is 0 and the maximum is 24.
f Calculated as distress for four items on the CAARMS positive symptom subscale, each ranging from 0 to 100.
g CAARMS symptoms: ‘social isolation’ and ‘impaired role function’ were not taken into account for the analyses because of their overlap with the SOFAS.
h To keep all β coefficients positive, MANSA scores were inverted by subtracting 84 (higher score indicates a lower level, e.g. an original score of 20 = 64).
i TAU + CBTuhr = treatment-as-usual plus cognitive behavioral therapy for UHR (experimental condition).
j To keep all β coefficients positive, SOFAS scores were inverted by subtracting 100 (higher score indicates a lower level, e.g. an original score of 40 = 60).
Transition to psychosis
In the present study, we analysed 185 subjects with at least one follow-up clinical evaluation. The mean observation period was 451.5 [standard error (s.e.) = 11.8, median 540.0] days. During the 18 months follow-up, 32 subjects made a transition to psychosis. The instantaneous incidence rate (iIR), i.e. the HR, of transition to psychosis after 6, 12 and 18 months was 0.054, 0.130 and 0.173, respectively. The mean time to transition from baseline examination was 491.6 [s.e. = 10.2, 95% confidence interval (CI) = 471.67–511.51] days.
Prediction model
The final predictor model, adjusting for recruiting strategy, included five variables: CAARMS-Observed intensity of blunted affect (any score >2), CAARMS-Intensity of subjective complaints of impaired motor function (any score >2), PBIQ-R subjectively perceived social marginalization score (any score > 15), declined social functioning (as assessed with the SOFAS), and distress associated with non-bizarre ideas. The resulting equation was [1.220 × CAARMS-Observed intensity of blunted affect ⩾3] + [0.999 × CAARMS-Intensity of subjective complaints of impaired motor function ⩾3] + [1.017 × PBIQ-R social marginalization score ⩾16] + [0.084 × (100 − SOFAS score –54.0486)] + [0.017 × (CAARMS-distress associated with non-bizarre ideas-54.87)]. The SOFAS score was inverted by subtracting 100. The continuous covariates were centered around the mean, to make survival functions relative to the mean of continuous variables rather than relative to the minimum (Hosmer et al. Reference Hosmer, Lemeshow and May1999). A syntax for the regression formula can be obtained from the first author. The 3.388 HR emerging for the variable CAARMS-Observed intensity of blunted affect score of ⩾3 was the highest (Table 3), indicating that patients with a total subscale score of 3–6 transitioned to psychosis at a rate 3.388 times higher than patients with a lower score.
Table 3. Cox proportional hazard model
CAARMS, Comprehensive Assessment of At-Risk Mental States; CI, confidence interval; HR, hazard ratio; PBIQ-R, Personal Beliefs about Illness Questionnaire; SOFAS, Social and Occupational Functioning Assessment Scale.
a To keep all β coefficients positive, SOFAS scores have been inverted by subtracting 100 (higher score indicates lower level, e.g. an original score of 40 = 60).
The small number of transitions to psychosis precluded splitting to generate a developing and a validation sample. Therefore, we applied a bootstrap procedure generating 1000 samples for testing the robustness of the developed prediction model. The bootstrapped results are: CAARMS-Observed blunted affect ⩾3, β = 1.22 (s.e. = 0.43, 95% CI 0.42–2.10, p = 0.001); CAARMS-Subjective complaints of impaired motor function intensity ⩾3, β = 0.999 (s.e. = 0.42, 95% CI 0.21–1.88, p = 0.01); PBIQ-R social marginalization, β = 1.017 (s.e. = 0.46, 95% CI 0.13–1.95, p = 0.01); SOFAS, β = 0.08 (s.e. = 0.04, 95% CI 0.02–0.17, p = 0.02); and CAARMS-distress associated with non-bizarre ideas β = 0.02 (s.e. = 0.01, 95% CI 0.01–0.04, p = 0.01). If the CI fails to include 0, then the p value is deemed to be ⩽0.05 and the effect is said to be significant (Loughin, Reference Loughin1998). Therefore, it can be concluded that all predictors in the Cox model were still significant and that our model had a good internal validity.
Effect of add-on CBT treatment
According to the state-of-the-art manual (Hosmer et al. Reference Hosmer, Lemeshow and May1999), the effect of CBT treatment (n = 88, 47.6%) on the predictor model was tested by adding the treatment variable in a second block to the model, resulting in β = 0.520 (s.e. = 0.42) and Wald = 1.57 with HR 1.68 (95% CI 0.75–3.80). All five variables of the initial predictor set (Table 3) continued to make a significant contribution to the equation, whereas CBT no longer made a significant contribution to the model in the final multivariate analysis without blocking (p = 0.21).
Prognostic index
Individual prognostic scores were calculated by applying the final Cox model to each subject. Based on the resulting prognostic score, a PI with three risk classes could be generated (Table 4).
Table 4. Risk classes of the prognostic index (n = 185)
CI, Confidence interval; iIR, instantaneous incidence rate; s.e., standard error.
a The prognostic score is calculated as [1.220 × CAARMS-Observed intensity of blunted affect ⩾3] + [0.999 × CAARMS-Intensity of subjective complaints of impaired motor function ⩾ 3] + [1.017 × PBIQ-R social marginalization score ⩾ 16] + [0.084 × (100 − SOFAS score -54.0486)] + [0.017 × (CAARMS-distress associated with non-bizarre ideas − 54.87)].
Fig. 1 shows the corresponding Kaplan–Meier survival curves for the three risk classes calculated on the total sample. The 18-month HRs for classes I, II and III were 0.060, 0.176, and 1.569, respectively. With regard to the survival curves, class I differed significantly from class III [χ2(1) = 57.26, p < 0.001]. Furthermore, class I differed significantly from class II [χ2(1) = 4.21, p < 0.04] and class II from class III [χ2(1) = 52.00, p < 0.001]. The mean time to transition in class III differed from class II by 215.5 days and from class I by 248.3 days, with a totally distinct 95% CI. Applying the prognostic index to the CBT and TAU subsamples separately produced comparable HRs for risk class I and II compared to the HR of the whole sample. The HR for risk class III of the TAU subsample was comparable to the HR of the whole sample (Table 4). Furthermore, the survival curves are distinct in both subsamples.
Fig. 1. Kaplan–Meier survival analysis for risk classes of prognostic index (n = 187); 18-month hazard rate: class I = 0.060, class II = 0.176, and class III = 1.569.
Prognostic accuracy measures
As prognostic accuracy measures cannot be analysed with censored data, our calculations had to rely on the subsample with a known state of transition at 18 months follow-up (n = 167; 90.3%). Of note, there was no difference in the prognostic scores between the subsamples with known or unknown state of transition (U = 1314.50, p = 0.38). In total 32 (19.2%, CBT 10 v. TAU 22) of the 167 patients (CBT 78 v. TAU 89) made a transition to psychosis within 18 months. In risk class III, 80.0% (CBT 2/78 = 2.6% v. TAU 14/89 = 15.7%) transitioned to psychosis within the same time-frame compared to 14.6% (CBT 7/78 = 9.0% v. TAU 7/89 = 7.9%) in risk class II and 3.9% (CBT 1/78 = 1.3% v. TAU 1/89 = 1.1%) in risk class I.
Entering the prognostic scores of the whole sample in the binary logistic regression with the default probability threshold of 0.50, sensitivity was 0.38; specificity was 0.98, with a PPV = 0.80 and a NPV = 0.87. The +LR was 19.0 and the −LR was 0.63. The overall accuracy of the model was 86.2%. The AUC of the EDIE prediction model was 0.81 (s.e. = 4.4, 95% CI 71.90–89.20, p < 0.001).
Discussion
Our results indicate that a predictor model consisting of five baseline variables (observed blunted affect, subjective complaints of impaired motor function, subjectively experienced social marginalization, decline in social functioning, and distress associated with non-bizarre ideas) can significantly improve accuracy in predicting future psychosis in a help-seeking UHR stage-1b sample of patients with low functioning. From a clinical viewpoint, this stage is most important because the ‘caseness’ demands therapy for persons at imminent risk of psychosis (McGorry et al. Reference McGorry, Hickie, Yung, Pantelis and Jackson2006; Yung et al. Reference Yung, Stanford, Cosgrave, Killackey, Phillips, Nelson and McGorry2006). The AUC of the EDIE prediction model showed an excellent ability to discriminate between transition and non-transition. The +LR of 16.9 indicates that a positive classification makes it 17 times more likely to develop a psychosis than a negative test. The PPV of 80.0% also indicates that the model, which is comparable to that used in other studies (Cannon et al. Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker, Seidman, Perkins, Tsuang, McGlashan and Heinssen2008; Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ; Nieman et al. Reference Nieman, Ruhrmann, Dragt, Soen, van Tricht, Koelman, Bour, Velthorst, Becker, Weiser, Linszen and de Haan2014; Cornblatt et al. Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen, John and Correll2015), has an excellent value for correct detection of a person at risk if it was only used for dichotomous classification. However, comparable to the EPOS predictive model (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ), the high −LR of 19.0 demonstrates that the model is not suitable to rule out an increased risk of psychosis in patients with a negative test result. Furthermore, similar to the NAPLS (Cannon et al. Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker, Seidman, Perkins, Tsuang, McGlashan and Heinssen2008) and the EPOS prediction model (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ), our prognostic model showed a very low sensitivity, denying individuals at risk but scoring below threshold to access early intervention programs. Classification of individual prognostic scores to estimate the current risk of transition to psychosis is known to be a good alternative to overcome this problem with no loss of sensitivity (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ; Nieman et al. Reference Nieman, Ruhrmann, Dragt, Soen, van Tricht, Koelman, Bour, Velthorst, Becker, Weiser, Linszen and de Haan2014).
Personalized risk estimation
Based on the prognostic scores of the five-variable EDIE-NL prediction model, i.e. the Cox regression equation, we identified three statistically distinct risk classes that are able to further classify the magnitude of the psychosis risk in selected risk groups.
The ilR in the highest class was almost 26 times higher than that in the lowest class, and >9 times higher than that in class II. Thus, compared with the 18-month general HR of 0.173 predicted by the UHR inclusion criteria, applying our model as a second step of risk stratification led to an important improvement in individual risk assessment. This included the ability to predict not only the magnitude of risk but also the time to transition, which differed markedly between class III and the other classes; the mean difference compared with the lowest class was ⩾8 months. In addition, in the sample with known outcome (n = 167), in the lowest class 3.9% of the subjects transitioned within 18 months, while in the highest risk class, about 80.0% transitioned within this time-frame. Therefore, the different risk classes may be useful to healthcare professionals to stratify and personalize treatment. For example, low index scores could be interpreted as indicating minimal risk, with little treatment necessary, i.e. monitoring of at-risk symptoms twice per year could be sufficient. High index scores could suggest CBT with additional outreach systemic treatment (i.e. talking to school teachers or coaching in the work situation, help with social relations and family support) to prevent withdrawal and exclusion. The future challenge is develop and test adequate interventions for each stratum of risk.
Predictors
All five predictors included in the EDIE prediction model have been linked to psychosis and are thus clinically meaningful. Two of the predictors found in the present study appear to be attenuated negative symptoms of psychosis, i.e. flattened affect and motor abnormalities. It is known that attenuated negative symptoms are an important part of the UHR status (Cornblatt et al. Reference Cornblatt, Lencz and Obuchowski2002; Nelson et al. Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner, Broussard, Simmons, Foley, Brewer, Francey, Amminger, Thompson, McGorry and Yung2013; Valmaggia et al. Reference Valmaggia, Stahl, Yung, Nelson, Fusar-Poli, McGorry and McGuire2013).
The association between a higher degree of restricted affect and an increased risk of transition to psychosis confirms the results of a previous study (Mason et al. Reference Mason, Startup, Halpin, Schall, Conrad and Carr2004) and appeared to correspond to one of the nine prodromal symptoms specified in the DSM-III-R (APA, 1987) which did not, however, become part of the UHR criteria.
Another predictor variable was subjectively experienced impairment in motor function in the absence of any detectable behavioural abnormality. This was also one of the prodromal symptoms in DSM-III-R (APA, 1987). Other researchers have also reported on disturbances in subjective motor functioning (Valmaggia et al. Reference Valmaggia, Stahl, Yung, Nelson, Fusar-Poli, McGorry and McGuire2013; Manschreck et al. Reference Manschreck, Chun, Merrill, Maher, Boshes, Glatt, Faraone, Tsuang and Seidman2015). Furthermore, subjective complaints of impaired motor functioning are also known as a basic symptom. Basic symptoms are defined as subtle, self-experienced, self-reported deficits that often remain solely in the self-perception of the patient and do not show in behaviour (Huber, Reference Huber1983; Schultze-Lutter, Reference Schultze-Lutter2009). The basic symptoms are included in the CAARMS (Yung et al. Reference Yung, Yuen, McGorry, Phillips, Kelly, Dell'Olio, Francey, Cosgrave, Killackey, Stanford, Godfrey and Buckby2005) and in the cognitive disturbances (COGDIS); both were previously found to be associated with high transition rates in UHR (Schultze-Lutter et al. Reference Schultze-Lutter, Ruhrmann, Picker, von Reventlow, Brockhaus-Dumke and Klosterkötter2007).
Next, the predictor variable PBIQ-R item subjectively experienced social marginalization is characterized by a person's cognitive appraisal of social participation. Thus, how a person attributes meaning to social disadvantage is linked to a higher risk of conversion.
The inclusion of SOFAS deterioration in social and role functioning as a predictor corresponds with findings showing that transition to first-episode psychosis is associated with reduced levels of social functioning (Yung et al. Reference Yung, Phillips, Yuen, Mcgorry and Pan2004, Reference Yung, Stanford, Cosgrave, Killackey, Phillips, Nelson and McGorry2006; Addington et al. Reference Addington, Penn, Woods, Addington and Perkins2008; Cannon et al. Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker, Seidman, Perkins, Tsuang, McGlashan and Heinssen2008; Fusar-Poli et al. Reference Fusar-Poli, Meneghelli, Valmaggia, Allen, Galvan, McGuire and Cocchi2009, Reference Fusar-Poli, Byrne, Valmaggia, Day, Tabraham, Johns and McGuire2010; Riecher-Rössler et al. Reference Riecher-Rössler, Pflueger, Aston, Borgwardt, Brewer, Gschwandtner and Stieglitz2009; Velthorst et al. Reference Velthorst, Nieman, Linszen, Becker, de Haan, Dingemans, Birchwood, Patterson, Salokangas, Heinimaa, Heinz, Juckel, von Reventlow, French, Stevens, Schultze-Lutter, Klosterkötter and Ruhrmann2010; Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ; Cornblatt et al. Reference Cornblatt, Carrión, Addington, Seidman, Walker, Cannon, Cadenhead, McGlashan, Perkins, Tsuang, Woods, Heinssen and Lencz2012, Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen, John and Correll2015; Demjaha et al. Reference Demjaha, Valmaggia, Stahl, Byrne and McGuire2012; Valmaggia et al. Reference Valmaggia, Stahl, Yung, Nelson, Fusar-Poli, McGorry and McGuire2013; Nelson et al. Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner, Broussard, Simmons, Foley, Brewer, Francey, Amminger, Thompson, McGorry and Yung2013). This is an interesting finding as the SOFAS criterion is already included in the UHR criteria. One possible explanation is that, independent of conversion to psychosis, patients may still experience significant levels of impaired social functioning (Morrison et al. Reference Morrison, French, Parker, Roberts, Stevens, Bentall and Lewis2007; Yung et al. Reference Yung, Buckby, Cosgrave, Killackey, Baker, Cotton and McGorry2007); however, those patients that converted may have much poorer social functioning.
The association between a higher distress score associated with the CAARMS item non-bizarre ideas, and an increased risk of psychosis, corresponds with the results of recent studies. Both the NAPLS and the EPOS reported that suspicion predicted conversion (Cannon et al. Reference Cannon, Cadenhead, Cornblatt, Woods, Addington, Walker, Seidman, Perkins, Tsuang, McGlashan and Heinssen2008; Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ). Another study showed that subclinical psychotic symptoms with distress and (eventually) with help-seeking behaviour are more clinically relevant and have a higher risk for conversion (van Os et al. Reference van Os, Linscott, Myin-Germeys, Delespaul, Krabbendam and Van Os2009; Brett et al. Reference Brett, Heriot-Maitland, McGuire and Peters2014). In contrast to our findings, i.e. that the distress associated with non-bizarre ideas is a predictor of psychosis, the study of Power et al. Reference Power, Polari, Yung, McGorry and Nelson2015 did not find such an association (Power et al. Reference Power, Polari, Yung, McGorry and Nelson2015). One point of consideration is that perceptual abnormalities were rated the most distressing in their younger sample (18.5 years) compared to our older sample (22.7 years). Perceptual aberrations are prevalent in adolescence, but also highly transient. In our somewhat older sample, secondary delusions on the origin and power of auditory hallucinations may explain the higher distress (Krabbendam et al. Reference Krabbendam, Myin-Germeys, Hanssen, Bijl, de Graaf, Vollebergh, Bak and van Os2004).
In contrast to the NAPLS and the EPOS finding on positive symptoms, neither the total number of positive symptoms nor a specific positive symptom class were retained in the EDIE-NL prediction model. It has been shown that negative rather than positive symptoms can have a significant impact on the transition from a UHR state to a full-blown psychosis (Lencz et al. Reference Lencz, Smith, Auther, Correll and Cornblatt2004; Yung et al. Reference Yung, Nelson, Thompson and Wood2010; Velthorst et al. Reference Velthorst, Nieman, Klaassen, Becker, Dingemans, Linszen and De Haan2011; Demjaha et al. Reference Demjaha, Valmaggia, Stahl, Byrne and McGuire2012; Valmaggia et al. Reference Valmaggia, Stahl, Yung, Nelson, Fusar-Poli, McGorry and McGuire2013). In fact, positive symptoms may be transient and often remit without any treatment within 1 year from first presentation (Cornblatt et al. Reference Cornblatt, Lencz, Smith, Correll, Auther and Nakayama2003; van Os et al. Reference van Os, Linscott, Myin-Germeys, Delespaul, Krabbendam and Van Os2009; Simon & Umbricht, Reference Simon and Umbricht2010; Yung et al. Reference Yung, Nelson, Thompson and Wood2010; Velthorst et al. Reference Velthorst, Nieman, Klaassen, Becker, Dingemans, Linszen and De Haan2011). It is important to note, however, that the EDIE-NL comprised a stage-dependent sample of only stage-1b UHR patients, recruited via a two-step screening procedure in secondary mental healthcare (Ising et al. Reference Ising, Veling, Loewy, Rietveld, Rietdijk, Dragt, Klaassen, Nieman, Wunderink, Linszen and van der Gaag2012). Therefore, the variance in positive symptoms between patients is small and lacks predictive power. In samples with more diverse characteristics, however, the symptoms may well have predictive value.
Furthermore, based on this study and other studies (EPOS, NAPLS, PACE400) it appears that a strong PPV can be achieved using psychopathological data alone, which should therefore (at the moment) be preferred to extensive neuroimaging batteries.
Effect of treatment
The EDIE-NL trial (Rietdijk et al. Reference Rietdijk, Dragt, Klaassen, Ising, Nieman, Wunderink, Delespaul, Cuijpers, Linszen and van der Gaag2010) was an RCT designed to study the benefits of add-on CBT, targeted at the prevention of psychosis in a help-seeking UHR population. The EDIE-NL demonstrated a statistically significant risk reduction with CBT of about 50% (van der Gaag et al. Reference van der Gaag, Nieman, Rietdijk, Dragt, Ising, Klaassen, Koeter, Cuijpers, Wunderink and Linszen2012). Following Hosmer et al. (Reference Hosmer, Lemeshow and May1999), in the final prediction model we added treatment as a potential confounder. Treatment showed a HR of 1.683 but had no significant additive predictive value. Because the predictive model was equally strong in the experimental and control group, this implies that the effect of CBT on reducing the transition rate did not operate via targeting these particular risk factors, but through some other mechanism (such as changing the appraisal of beginning positive symptoms to prevent delusional explanations).
Strengths
The strengths of this study are the large number of participants, the low dropout rate, and the precise and repetitive assessment of transition to psychosis. Because the predictors were assessed in clinical practice, the detection of UHR patients can probably be improved in clinical practice. Furthermore, the patients can be classified into different risk classes.
Limitations
The limited number of transitions did not allow to split the sample to validate the model. However, bootstrapping was used to assess the internal validity of the Cox model.
The final predictors were derived by a statistical approach, by screening potential predictors for association with conversion to full-blown psychosis in multivariate models within each assessment domain, and only those predictors that contributed uniquely to conversion in an overall (cross-domain) multivariate model were retained. This approach could lead to an overfitting. However, to circumvent this risk, and to check the internal validity, a bootstrap analysis was used. Nevertheless, these predictors should be replicated in an independent study with similar features.
Several studies have found duration of symptoms to be a strong predictor of transition to psychosis (Nelson et al. Reference Nelson, Yuen, Wood, Lin, Spiliotacopoulos, Bruxner, Broussard, Simmons, Foley, Brewer, Francey, Amminger, Thompson, McGorry and Yung2013). However, symptom duration was not measured in the current study.
In this trial, about 19% of the patients identified with an UHR were not willing to participate (van der Gaag et al. Reference van der Gaag, Nieman, Rietdijk, Dragt, Ising, Klaassen, Koeter, Cuijpers, Wunderink and Linszen2012). In this sense, patients who consent to participate in a RCT may differ in a substantial way from other samples. In the present study the identified predictors are conditional on our UHR sample of help-seeking persons with co-morbid disorders who, in addition, were willing to participate in a trial.
Conclusion
Our results suggest that predicting a first-episode psychosis in UHR patients is improved using a five-predictor stage 1b-dependent prognostic model, including negative symptoms (observed flattened affect, subjective impaired motor functioning), impaired social functioning, beliefs about social marginalization and distress associated with non-bizarre ideas in a two-step algorithm combining risk detection and stratification. When translating our findings to clinical practice, and in line with others (Ruhrmann et al. Reference Ruhrmann, Schultze-Lutter, Salokangas, Heinimaa, Linszen, Dingemans, Birchwood, Patterson, Juckel, Heinz, Morrison, Lewis, von Reventlow and Klosterkötter2010b ; Michel et al. Reference Michel, Ruhrmann, Schimmelmann, Klosterkotter and Schultze-Lutter2014; Cornblatt et al. Reference Cornblatt, Carrión, Auther, McLaughlin, Olsen, John and Correll2015), negative symptoms and social functioning should be incorporated in future UHR criteria to advance psychosis prediction.
Supplementary material
For supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S0033291716000325.
Acknowledgements
This study was funded by the Netherlands Organization for Health Research and Development, ZonMW (grant no. 120510001). ZonMW had no further role in the study design, collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication. The authors gratefully acknowledge the contribution of all participants, research assistants, therapists and all others who took part or contributed to the EDIE-NL study. We also thank Marion Bruns for her organizational contributions to the study.
Declaration of Interest
None.
