Effective prediction of clinical high risk of psychosis (CHR) is a current major challenge for the psychiatric medical community and has seen little progress over the last several years (in general, accuracy rates remain under 75%). One of the most important risk calculators for predicting conversion to psychosis in CHR individuals is available on the internet (http://riskcalc.org:3838/napls/), and based on clinical, cognitive, and demographic data from the NAPLS-2 sample (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016). Our previously study cross-validated the NAPLS-2 psychosis risk calculator in a Chinese CHR sample (Zhang et al., Reference Zhang, Li, Tang, Niznikiewicz, Shenton, Keshavan, Stone, McCarley, Seidman and Wang2018a). However, a fundamental question remains, whether conversion is the appropriate outcome to be predicted.
The ‘bad’ outcome in prediction studies has generally been defined as conversion to psychosis, which is in accordance with the criteria of the ‘Presence of a Psychotic Syndrome (POPS)’ (McGlashan et al., Reference McGlashan, Walsh and Woods2010). The operational definition of conversion was based on the presence of a six-level positive symptom scale (the rating ‘6’ refers to severe and psychotic, i.e. conviction of psychotic experiences). However, such a specific definition of conversion based only on positive symptoms may not be the best option (van Os and Guloksuz, Reference van Os and Guloksuz2017). From a clinical point of view, currently defined endpoints may neither adequately address the characteristic presenting symptoms (such as the negative symptoms) and functioning (such as social and role functioning and cognitive deficits) during the progression of the disease nor reflect the clinicians' and patients' concerns (Zhang et al., Reference Zhang, Xu, Tang, Cui, Li, Wei, Xu, Jiang, Zhu, Li, Jiang, Xiao and Wang2018b). Clinically it is extremely important to know whether a poor functional outcome is also predictable when using risk calculators in CHR individuals. Several non-drug interventions such as supportive psychotherapy (Rosenbaum et al., Reference Rosenbaum, Harder, Knudsen, Koster, Lindhardt, Lajer, Valbak and Winther2012) are actually more about improving global function than treating psychotic symptoms. Since there are broad debates (Miller et al., Reference Miller, McGlashan, Rosen, Somjee, Markovich, Stein and Woods2002; Liu and Demjaha, Reference Liu and Demjaha2013) on antipsychotic treatment for CHR individuals, if functional outcomes can be predicted and become the target of treatment, these non-drug interventions may play a more important role in early intervention. In this context, we provide evidence based on a recent and ongoing project with a large CHR cohort, to emphasize the needs for an extended definition of CHR outcomes to improve the precision and value of predictions.
Methods
Project
The SHARP study was conducted with CHR individuals enrolled in an early identification program for psychosis, implemented at one site, the SMHC in China. The Research Ethics Committees at the SMHC approved these studies. A key element of the SHARP study is that all participants are psychotropically naïve when they enter the study and are assessed clinically. They generally have had no treatment of any kind for a psychiatric disorder. There is also no history of substance abuse or dependence. Participants are not treated in the study, but receive treatment as required, provided by their community psychiatrist, after their baseline assessment. As noted above, 300 CHR individuals were recruited and assessed during 2012–2016 (Fig. 1).
Fig. 1. Flow chart of sample recruitment and follow-up. Note: Overview of the Shanghai at Risk for Psychosis (SHARP) program: (1) first phase of the SHARP program recruited 100 CHR youths between August 2012 and May 2014, supported by a Fogarty and National Institutes of Mental Health (NIMH) grant (1R21 MH093294-01A1), titled, ‘Broadening the Investigation of Psychosis Prodrome to Different Cultural Groups’ (abbreviated to R21). (2) Second phase of the SHARP program recruited 200 CHR youths between June 2014 and January 2017, supported by an NIMH grant (1R01 MH 101052-01), titled ‘Validating Biomarkers for the Prodrome and Transition to Psychosis in Shanghai’ (abbreviated to R01). (3) Third phase of the SHARP program is ongoing with the second R01 NIMH grant (1 R01 MH111448-01) titled ‘A Psychobiological Follow-up Study of Transition from Prodrome to Early Psychosis’.
Sample
All participants gave written informed consent at the recruitment stage of the study. Subjects younger than 18 years of age had their consent forms signed by their parents and provided assent personally. A total of 300 CHR individuals were identified in a face-to-face interview using the SIPS (Miller et al., Reference Miller, McGlashan, Rosen, Somjee, Markovich, Stein and Woods2002, Reference Miller, McGlashan, Rosen, Cadenhead, Cannon, Ventura, McFarlane, Perkins, Pearlson and Woods2003). Among them, 228 (76.0%) completed neurocognitive assessments at baseline. There was no difference between subjects with and without neurocognitive assessments on the NAPLS-2 key predictor variables except for age. Subjects with cognitive assessments [mean age 19.0 years (s.d. = 5.0), n = 228] were younger than those without cognitive assessments [mean age 20.9 (s.d. = 6.4), n = 72]. Of the 228 subjects with neurocognitive assessments, 199 (87.3%) CHR individuals had at least a 1-year follow-up assessment. At baseline, subjects without follow-up assessments (n = 29) had higher levels of unusual thought content and suspiciousness [mean score, 4.7 (s.d. = 1.3) v. 3.1 (s.d. = 1.5), t = 5.424, p < 0.001], and a greater decline in general functioning [3.5 (s.d. = 0.8) v. 3.1 (s.d. = 0.9), t = 2.094, p = 0.037] than those with follow-up assessments. Finally, the data from the 199 CHR individuals were used as the validation sample for testing the NAPLS-2 psychosis risk calculator. Details of the study procedures, study setting, implementation of the measurement and assessment are reported elsewhere (Zheng et al., Reference Zheng, Wang, Zhang, Li, Li and Jiang2012; Zhang et al., Reference Zhang, Li, Woodberry, Seidman, Zheng, Li, Zhao, Tang, Guo, Lu, Zhuo, Qian, Chow, Li, Jiang, Xiao and Wang2014; Zhang et al., Reference Zhang, Li, Woodberry, Seidman, Chow, Xiao and Wang2015; Zhang et al., Reference Zhang, Li, Woodberry, Xu, Tang, Guo, Cui, Liu, Chow, Li, Jiang, Xiao, Seidman and Wang2017).
Predictors
There were eight predictor variables included in the NAPLS-2 psychosis risk calculator in Table 1.
Table 1. Predictor variables included in the NAPLS-2 and SHARP model
Outcome variables
Of the total 199 CHR individuals, 46 (23.1%) converted to full psychosis over 2 years of follow-up. Conversion to psychosis was defined using the POPS criteria (McGlashan et al., Reference McGlashan, Walsh and Woods2010).
In addition to the outcome categories of converters and non-converters, we divided the existing sample into two broad categories: recovered and non-recovered (Fig. 2). We contend that a category of ‘non-recovered' representing a poor functional outcome despite medical treatment is a helpful designation for functional outcomes to take into account the potential ameliorating effects of antipsychotics on positive symptoms. ‘Non-recovered' included two subgroups: (1) converters and (2) medicated CHR individuals with either unremitting positive symptoms (but which did not reach the level of persistent psychosis) or poor global functioning at follow-up called ‘treatment refractory’. The operational definition of ‘treatment refractory’ included treatment criteria and refractory criteria in two parts: (1) the treatment criteria (medication was almost the only treatment administered in this sample) was defined either as having taken higher therapeutic dosages of antipsychotic medicines (equivalent to ⩾3 mg/day risperidone) for at least 4 consecutive weeks, or lower therapeutic dosages (equivalent to <3 mg/day risperidone) for at least 6 consecutive months; and, (2) refractory criterion: unremitting symptoms or poor global functioning. The unremitting symptoms were defined as scores of 3–6 for positive symptoms in the SIPS (including Brief Intermittent Psychotic Symptoms/BIPS) at the follow-up visit. Poor global functioning was defined as a current GAF score of less than 60 at the follow-up point (Austin et al., Reference Austin, Mors, Secher, Hjorthoj, Albert, Bertelsen, Jensen, Jeppesen, Petersen, Randers, Thorup and Nordentoft2013; Simonsen et al., Reference Simonsen, Faerden, Romm, Berg, Bjella, Sundet, Ueland, Andreassen and Melle2017). This subgroup represents a type of atypical conversion that CHR individuals' positive symptoms are suppressed with antipsychotic treatment, but with poor outcomes even though their condition has not progressed completely to psychosis. The remaining CHR individuals were classified as ‘recovered’.
Fig. 2. Outcome classification chart.
In our previous investigation (Zhang et al., Reference Zhang, Li, Woodberry, Xu, Tang, Guo, Cui, Liu, Chow, Li, Jiang, Xiao, Seidman and Wang2017), we observed that compared to the NAPLS sample, a substantially higher percentage of participants in the SHARP sample were prescribed antipsychotics after entering the study rather than before, i.e. after their clinical and cognitive assessments were completed. In the current study, among the final sample of 199 CHR individuals, 160 (80.4%) had taken antipsychotics, 41 (20.6%) had taken antidepressants, and only one subject had undergone four sessions of psychotherapy by the final follow-up. All of these treatments were administered by non-study psychiatrists in the community after baseline clinical and neurocognitive assessment.
Procedures
Participants who met the inclusion criteria were consecutively referred by our clinicians. The participants were informed that this part of the study involved a group of clinical, cognitive and biomarker assessments at baseline with a naturalistic follow-up every 6 months. They would follow the routine clinical treatment procedures provided at the SMHC. All participants from the first visit were followed up for at least 1 year once we attained their consent and intake evaluation information.
All CHR individuals who completed the baseline assessment were followed up every 6 months. Both the CHR individuals and their caregivers had been told that they could contact the interviewer and study clinicians anytime for questions and progress reports on the patients' medical conditions. Except for those who did not desire any further contact, the CHR participants were re-assessed by telephone at the 6th and 18th months or by face-to-face interview using the SIPS at the 12th and 24th months. The outcome determination was based mainly on the face-to-face interviews (of 199 CHR individuals, 119 had at least once face-to-face interview), partly from telephone interviews of CHR individuals or their caregivers, and on the clinical information obtained from clinician reports. The outcome measure included three major components: (1) current symptom level (SIPS interview); (2) current functional level (GAF assessment) and, (3) clinical treatment (detailed pharmacological and psychosocial therapy history).
Data analysis
SPSS version 16.0 was used for data analysis. The six key predictor variables were entered into the NAPLS-2 psychosis risk calculator by two people independently. A new variable of risk ratio for each CHR was constructed by the calculator. Receiver operating characteristic (ROC) analysis (Hanley and McNeil, Reference Hanley and McNeil1982) was used to test whether the new risk ratio distinguished between converters and non-converters (recovered and non-recovered). The predictive value of the NAPLS-2 psychosis risk calculator was determined according to the area under the ROC curve (AUC) (Swets, Reference Swets1988). The sensitivity, specificity, positive predictive value and negative predictive values of the NAPLS-2 calculator were examined across different levels of predicted risk. To evaluate the predictive value of the six key predictors from the NAPLS-2 calculator, we performed a binary logistic regression analysis to explore which predictors at baseline best predicted converters and non-converters. Similarly, we assessed recovered v. non-recovered subjects. The odds ratio and 95% confidence intervals were estimated for the risk of conversion to psychosis (and non-recovered) in this external validation sample. Based on this regression model, we generated the probabilities of risk for each case. The ROC methodology was used to assess the discriminative power of the probabilities.
Results
Predictor variables included in the risk calculator were compared between conversion v. non-conversion and recovered v. non-recovered CHR youths in the SHARP samples (Table 2). When comparing predictor variables between groups, the difference between non-recovered v. recovered was more distinct than the difference between conversion v. non-conversion.
Table 2. Baseline predictor variables, comparison between converters and non-converters, and recovered and non-recovered CHR subjects
BACS, Brief Assessment of Cognition in Schizophrenia; HVLT-R, Hopkins Verbal Learning Test–Revised; t/Z/χ2, t for independent t test, Z for Mann–Whitney U test (nonparametric test), χ2 for κ test.
We investigated whether probability risk estimates provided by the NAPLS-2 calculator for each individual in the SHARP validation sample could discriminate converters from non-converters, and recovered CHR subjects from non-recovered. Figure 3 shows that when using recovered or non-recovered as the endpoint (i.e. converters + non-converter treatment refractory = non-recovered), the ROC analysis resulted in an AUC of 0.754 (p < 0.001) for the probability risk estimates, which is better than the AUC for conversion as the principal endpoint (0.631, p = 0.007).
Fig. 3. Receiver operating characteristic curve for the SHARP (Shanghai At Risk for Psychosis project) validation model, classified based on conversion (conversion and non-conversion to psychosis were used as the outcome classification. Conversion was determined using the Structured Interview for Prodromal Symptoms (SIPS)] and non-recovered (converters and medicated CHR individuals with unremitting symptoms or poor global functioning were defined as non-recovered. Medication criteria were defined as either having taken high dosages of antipsychotic medicines for at least 4 weeks, or taken low dosages for at least 6 months. The unremitting symptoms criterion required subjects to have scores of 3–6 for positive symptom in the SIPS. Poor global functioning required subjects to have a current GAF score less than 60 at the follow-up point).
As shown in Table 3, a regression model was used to evaluate the effect of six key predictor variables used in the NAPLS-2 psychosis risk calculator in the prediction of poor functional outcome in the SHARP sample. The value of risk probabilities was generated using this regression model for each case and then used for ROC analysis. The overall model for predicting the outcome (either conversion or non-recovered) was not significant when all six independent variables were entered simultaneously, with an overall classification accuracy of 74.6–80.5%. In terms of individual variables, only the unusual thought content + suspiciousness/paranoia item and decline in function showed good discrimination in the model for predicting non-recovered CHR subjects.
Table 3. Performance of key predictors from the NAPLS-2 psychosis risk calculator in the SHARP sample
a ‘Modified P1 + P2 SIPS items’ represents the severity of unusual thought content and suspiciousness (items P1 and P2 in SIPS). The P1 or P2 item rated 0–2 on the original scale are recoded as 0, 3–6 on the original scale are recoded as 1–4.
b ‘Decline in global assessment of function (GAF): revised’ represents the social functioning decline, corresponding to the ‘Decline in Global Functioning: Social scale score’ from the NAPLS-2 calculator. The GAF score is dropped to 5% or less of previous best GAF are recoded as 0, dropped to 5–15%, 15–25%, 25–35%, 35–45%, 45–55%, 55–65% of previous best GAF are recorded as 1–6. Decline in GAF score: Change in GAF in the year prior to baseline was derived from the highest GAF score in the past year prior to baseline minus baseline GAF score.
c ‘AUC’: The receiver operating characteristic curve.
Discussion
Through validation of the NAPLS-2 risk calculator in a Chinese CHR sample in Shanghai, the major aim of this study was to evaluate and compare the accuracy of predicting conversion v. poor functional outcome. To the best of our knowledge, this is the first attempt to perform such a comparison. Regarding the prediction of conversion to psychosis, the NAPLS-2 calculator did not fit our SHARP data as well as it fit the NAPLS-2 sample; probability risk estimates yielded an AUC of 0.631 with moderate predictive power, but lower than the NAPLS-2 AUC (C-index = 0.71). However, in the prediction of poor functional outcome (‘non-recovered’, which includes the conversion subgroup), our data show more accurate outcomes (AUC = 0.754) than for conversion alone, comparable to findings reported in NAPLS-2. To be specific, for those CHR youths with risk calculator estimates higher than 30%, these estimates had moderate sensitivity (53%) and excellent specificity (86%) for predicting a poor functional outcome. This information provides a critical first step in the development of the risk calculator for predicting a poor functional outcome, rather than predicting conversion alone.
The ultimate goal for CHR identification is to provide more precise risk prediction to guide evidence-based, personalized treatments. However, no previous studies have explored whether outcomes other than conversion, such as poor functioning or treatment responses, are better predicted. Our data obviously show that a poor functional outcome is better predicted the conversion. This may be explained by the fact that conversion only considers the progression of positive psychotic symptoms, which can be paroxysmal. However, functional performance can continue to deteriorate, which can be more predictable. Another reason is those predictors (such as cognitive and general function) included in the risk calculator are more likely to be related to functional outcomes.
Consistent with the NAPLS-2 model and our previous findings, the baseline severity level of unusual thought content + suspiciousness and global function decline were significant predictors of psychosis and poor functional outcome in the SHARP sample. Of note here, evidence has been accumulating that baseline disordered thought symptoms and functional deterioration is key risk factors for predicting the onset of psychosis in CHR syndromes (Fusar-Poli et al., Reference Fusar-Poli, Borgwardt, Bechdolf, Addington, Riecher-Rossler, Schultze-Lutter, Keshavan, Wood, Ruhrmann, Seidman, Valmaggia, Cannon, Velthorst, De Haan, Cornblatt, Bonoldi, Birchwood, McGlashan, Carpenter, McGorry, Klosterkotter, McGuire and Yung2013). In addition, our data highlight the importance of a declining GAF score in the prediction of psychosis. We used this measure in place of the Global Functioning: Social scale used in the NAPLS-2 model because the latter has not been used by Chinese clinicians. Similarly, cultural differences have not been examined and may affect its validity. In addition, its predictive value in the EDIPPP model was not significant (Cornblatt et al., Reference Cornblatt, Carrion, Auther, McLaughlin, Olsen, John and Correll2015). In contrast, the GAF score can be derived from the SIPS assessment and has been widely used in China for many years. Therefore, our results further showed that the GAF could be an alternative measure for predictive models, especially for a predicting poor outcome.
Although scores on the BACS and HVLT-R neurocognitive tests were significantly different between recovered and non-recovered CHR groups, they are not significant predictors of conversion or non-recovery in the SHARP sample. One possible cause of this discrepancy is the difference in the cognitive characteristics between SHARP and NAPLS-2 samples. Mean HVLT-R scores in the SHARP sample were different from those in the NAPLS-2 sample, which cannot exclude the effect of differences in the reliability or validity of MATRICS tests across populations. Furthermore, in contrast to the NAPLS-2 studies, family history of a psychotic disorder was not a significant predictor of psychosis in the SHARP sample. As for a family history of psychosis, the lower proportion of family history-positive cases in the SHARP sample could also account for the lower predictive power. There is also the possibility that information regarding psychosis history may be easily overlooked or unreported (Milne et al., Reference Milne, Caspi, Crump, Poulton, Rutter, Sears and Moffitt2009) in the SHARP sample due to the lack of psychiatric resources available to older generations and the stigma attached to diagnostic labels (Roy et al., Reference Roy, Walsh and Kendler1996).
In contrast to the US samples, the CHR individuals from SHARP were typically administered antipsychotic medications after their first visit to clinicians. Since the definition of conversion to psychosis in the NAPLS-2 study mainly relies on positive psychotic symptoms, the decrease in positive symptoms by antipsychotics in CHR individuals may be a new type of outcome in this Chinese cohort, which is characterized by systematic treatment of antipsychotic drugs, with a poor functional outcome or being symptomatic at the end of follow-up, but not experiencing conversion. We classified these individuals as ‘treatment refractory’, which represents atypical conversion. The positive symptoms among such CHR individuals are not as severe such that they should be classified as psychosis. Given the predictive value of more severe positive symptoms, which also make one more likely to be prescribed antipsychotics, it may be that a number of these individuals would have converted had they not been on medications at the time of assessment or that they may be more likely to convert over a longer follow-up period. The non-recovered CHR individuals who were treated systemically with antipsychotics in our sample manifested a lower level of positive symptoms at follow-up, but this may be due to medication effects. This possibility is supported by significantly worse negative symptoms with poorer general social and role function at follow-up compared with baseline; the functional outcome is less likely to be affected by antipsychotics. Together with conversion, prediction of non-recovery can offer clinicians more useful information than conversion alone. This underscores the importance of more frequent follow-ups of this subgroup to monitor clinical progression and treatment response.
Strengths and limitations
A major strength of our study is that we used identical measurements to the NAPLS-2 that were available (SIPS & MCCB) and had validated Chinese versions. Another strength is that in contrast to the NAPLS-2 data, which were collected from eight sites (Addington et al., Reference Addington, Cadenhead, Cornblatt, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods, Addington and Cannon2012), the current SHARP sample was recruited by one team from one catchment area, which may be more advantageous for its homogeneity. A limitation of the study is that the SHARP CHR cohort was surveyed naturalistically. There were 145 CHR individuals treated with antipsychotic for at least 2 weeks during the follow-up period. Previous studies showed that patients with schizophrenia who did not receive antipsychotics had better long-term outcomes compared to treated patients (Harrow et al., Reference Harrow, Jobe and Faull2012). Therefore, it remains unknown whether the antipsychotic can affect the functional outcome. The various medications the participants took with varying compliance may have confounded the results of functional outcome assessments, thereby limiting the generalizability to CHR subjects who have not taken any medication. Besides, the widespread use of antipsychotics in the current study may lead to the baseline positive symptoms was no longer a strong predictor of conversion (see Table 3). Moreover, although only 29 CHR individuals were lost in follow-up, they demonstrated more severe positive symptoms and poorer functioning at baseline than those who completed follow-up, which could bias our results by underestimating the clinical severity of our sample. Finally, as emphasized by Cannon (Cannon et al., Reference Cannon, Yu, Addington, Bearden, Cadenhead, Cornblatt, Heinssen, Jeffries, Mathalon, McGlashan, Perkins, Seidman, Tsuang, Walker, Woods and Kattan2016) and Carrion (Carrion et al., Reference Carrion, Cornblatt, Burton, Tso, Auther, Adelsheim, Calkins, Carter, Niendam, Sale, Taylor and McFarlane2016), the risk calculator remains experimental. It should only be used in research settings with clinicians who have undergone rigorous SIPS training (SIPS scores being at the core of the model) at this point, and not yet used in general clinical settings with individuals until its clinical utility and properties are validated more firmly.
In summary, although the definition of conversion to psychosis in most CHR-based studies relies only on positive psychotic symptoms, our evidence indicates that functional recovery should be provided a far more central position with respect to the outcome prediction of CHR individuals. It is suggested that a new standardized form of clinical outcome combining functional deficits should at least be developed and applied in future studies. Together with conversion, prediction of the functional outcome can provide clinicians with more useful information than that obtained when considering conversion alone.
Acknowledgements
This study was supported by the Ministry of Science and Technology of China, National Key R&D Program of China (2016YFC1306803), National Natural Science Foundation of China (81671329, 81671332), Shanghai Key Laboratory of Psychotic Disorders (13dz2260500), Science and Technology Commission of Shanghai Municipality (19441907800), Shanghai Jiaotong University Foundation (ZH2018ZDB03), The Clinical Research Center at Shanghai Mental Health Center (CRC2018ZD01, CRC2018ZD04 and CRC2018YB01) and Shanghai Mental Health Center Foundation (2016-FX-01, 2017-TSXK-03). For the purpose of commemorate, Dr Larry J. Seidman passed away on 7 September 2017 and Dr Robert W. McCarley passed away on 27 May 2017. Both were founders and core members of the SHARP (ShangHai At Risk for Psychosis) project.
Conflict of interest
None.
