Hostname: page-component-745bb68f8f-v2bm5 Total loading time: 0 Render date: 2025-02-06T06:09:55.097Z Has data issue: false hasContentIssue false

Psychosis as a risk factor for suicidal thoughts and behaviors: a meta-analysis of longitudinal studies

Published online by Cambridge University Press:  14 August 2017

X. Huang*
Affiliation:
Department of Psychology, Florida State University, Tallahassee, FL, USA
K. R. Fox
Affiliation:
Department of Psychology, Harvard University, Cambridge, MA,USA
J. D. Ribeiro
Affiliation:
Department of Psychology, Florida State University, Tallahassee, FL, USA
J. C. Franklin
Affiliation:
Department of Psychology, Florida State University, Tallahassee, FL, USA
*
*Address for correspondence: X. Huang, Department of Psychology, Florida State University, 1107 W. Call St., Tallahassee, FL, USA. (Email: huang@psy.fsu.edu)
Rights & Permissions [Opens in a new window]

Abstract

Background

Research has long noted higher prevalence rates of suicidal thoughts and behaviors among individuals with psychotic symptoms. Major theories have proposed several explanations to account for this association. Given the differences in the literature regarding the operationalization of psychosis and sample characteristics, a quantitative review is needed to determine to what extent and how psychosis confers risk for suicidality.

Methods

We searched PsycInfo, PubMed, and GoogleScholar for studies published before 1 January 2016. To be included in the analysis, studies must have used at least one psychosis-related factor to longitudinally predict suicide ideation, attempt, or death. The initial search yielded 2541 studies. Fifty studies were retained for analysis, yielding 128 statistical tests.

Results

Suicide death was the most commonly studied outcome (43.0%), followed by attempt (39.1%) and ideation (18.0%). The median follow-up length was 7.5 years. Overall, psychosis significantly conferred risk across three outcomes, with weighted mean ORs of 1.70 (1.39–2.08) for ideation, 1.36 (1.25–1.48) for attempt, and 1.40 (1.14–1.72) for death. Detailed analyses indicated that positive symptoms consistently conferred risk across outcomes; negative symptoms were not significantly associated with ideation, and were protective against death. Some small moderator effects were detected for sample characteristics.

Conclusions

Psychosis is a significant risk factor for suicide ideation, attempt, and death. The finding that positive symptoms increased suicide risk and negative symptoms seemed to decrease risk sheds light on the potential mechanisms for the association between psychosis and suicidality. We note several limitations of the literature and offer suggestions for future directions.

Type
Original Articles
Copyright
Copyright © Cambridge University Press 2017 

Introduction

Researchers and clinicians have long noted a strong association between psychosis and suicidal thoughts and behaviors (STBs). Approximately 30–40% of people with psychotic spectrum disorders think about suicide, 20–30% make a suicide attempt, and 5–10% die by suicide (Radomsky et al. Reference Radomsky, Haas, Keshavan and Sweeney1995; Palmer et al. Reference Palmer, Pankratz and Bostwick2005; Fialko et al. Reference Fialko, Freeman, Bebbington, Kuipers, Garety, Dunn and Fowler2006). These rates are alarmingly high considering that the 12-month prevalence rates of suicide ideation and attempt are about 3% and 0.5%, respectively, and that the rate of suicide death is 0.013% in the USA (Kessler et al. Reference Kessler, Berglund, Nock, Wang and Page2005; Kochanek et al. Reference Kochanek, Murphy, Xu and Tejada-Vera2016). Although reasons for this elevated risk remain unclear, multiple mutually non-exclusive explanations exist. For instance, psychotic symptoms (e.g. command hallucinations for suicide) might directly prompt STBs. Alternatively, psychosis may indirectly increase risk through concomitant effects of the illness such as feeling like a burden on others. Clarifying the role of psychosis in STBs is crucial for prediction and treatment. Therefore, this study aims to summarize existing literature to determine whether and how psychosis confers risk for STBs.

Before further discussion on the role of psychosis in suicidality, it is important to understand the term risk factor. According to Kraemer et al. (Reference Kraemer, Kazdin, Offord, Kessler, Jensen and Kupfer1997), a correlate is a factor associated with the outcome of interest, but the directionality of the association is unclear. Risk factors are special kinds of correlates that precede the outcome and divide the population into high- and low-risk groups (e.g. psychosis at Time 1 predicts suicide outcomes at Time 2). Cross-sectional studies can only establish correlates; longitudinal designs are necessary to identify risk factors. Thus, the effect of psychosis as a risk factor should be examined in the context of longitudinal studies.

It has been well documented that psychosis might directly confer risk for STBs via symptoms such as hallucinations and delusions (Koeda et al. Reference Koeda, Otsuka, Nakamura, Yambe, Fukumoto, Onuma, Saga, Yoshioka, Mita, Mizugai, Sakai and Endo2012). Past research suggests that command hallucinations for suicide are positively associated with suicide ideation and attempt (Kasper et al. Reference Kasper, Rogers and Adams1996; Harkavy-Friedman et al. Reference Harkavy-Friedman, Kimhy, Nelson, Venarde, Malaspina and Mann2003; Wong et al. Reference Wong, Öngür, Cohen, Ravichandran, Noam and Murphy2013). Hallucinations that are persistent, highly realistic, and congruent with delusional beliefs tend to incite a greater need to comply (Shawyer et al. Reference Shawyer, Mackinnon, Farhall, Sims, Blaney, Yardley, Daly, Mullen and Copolov2008). It is also likely that some types of delusions alone might prompt individuals to think about suicide and even take actions. For example, delusions that one is responsible for a crime or disaster might lead one to conclude that he or she deserves to die (Grunebaum et al. Reference Grunebaum, Oquendo, Harkavy-Friedman, Ellis, Li, Haas, Malone and Mann2001). Under this hypothesis, positive psychotic symptoms, but not negative symptoms, should be strong risk factors for STBs.

Indirect effects of psychosis are also plausible. For instance, the hopelessness theory posits that the cognitive style of making negative, stable, and global attributions about life events, the world, and self, is a sufficient cause for suicidality (Beck, Reference Beck1986; Abramson et al. Reference Abramson, Metalsky and Alloy1989, Reference Abramson, Alloy, Hogan, Whitehouse, Gibb, Hankin, Cornette, Joiner and Rudd2000). Disorders with psychotic symptoms such as schizophrenia are often associated with cognitive and memory impairment (Aleman et al. Reference Aleman, Hijman, De Haan and Kahn1999; Kuperberg & Heckers, Reference Kuperberg and Heckers2000), which predicts functional and economic loss (Green et al. Reference Green, Kern and Heaton2004; Marwaha & Johnson, Reference Marwaha and Johnson2004; Lewine, Reference Lewine2005). Individuals suffering from these symptoms and impairments might experience elevated hopelessness, which might in turn increase suicidality (Eisenberg & Lazarsfeld, Reference Eisenberg and Lazarsfeld1938; Kim et al. Reference Kim, Jayathilake and Meltzer2003; Lewine, Reference Lewine2005; Paul & Moser, Reference Paul and Moser2009). According to this theory, the global presence and severity of psychotic symptoms and disorders should be strong risk factors for STBs.

The interpersonal theory of suicide also suggests that psychosis might confer risk for suicide indirectly (Joiner, Reference Joiner2005; Van Orden et al. Reference Van Orden, Witte, Cukrowicz, Braithwaite, Selby and Joiner2010). This theory proposes that both the desire to die (i.e. perceived burdensomeness and thwarted belongingness) and the capability for suicide, defined as lowered fear of death and increased physical pain tolerance, must be present for someone to act on their suicidal intent. Psychosis is associated with an increased burden on caregivers (Awad & Voruganti, Reference Awad and Voruganti2008; Papastavrou et al. Reference Papastavrou, Charalambous, Tsangari and Karayiannis2012), and a diminished social connection due to impairment and stigma (Liddle, Reference Liddle2000; Lee et al. Reference Lee, Lee, Chiu and Kleinman2005). Psychosis might also increase acquired capability for suicide by habituating people to painful events such as homelessness (Folsom & Jeste, Reference Folsom and Jeste2002) and self-mutilation (Large et al. Reference Large, Babidge, Andrews, Storey and Nielssen2009). Based on this theory, the global presence and severity of psychosis would increase suicide ideation via perceived burdensomeness and thwarted belongingness; positive symptoms such as delusions and hallucinations would increase suicidal behaviors via increased capability for suicide resulted from experiences of painful events.

Despite these plausible hypotheses, mixed evidence in the field precluded a qualitative review of the literature from ascertaining the strength and the patterns of the association between psychosis and suicidality. Some studies reported higher prevalence rates of suicide thoughts and behaviors in psychotic populations (Kontaxakis et al. Reference Kontaxakis, Havaki-Kontaxaki, Margariti, Stamouli, Kollias and Christodoulou2004; Hawton et al. Reference Hawton, Sutton, Haw, Sinclair and Deeks2005; Palmer et al. Reference Palmer, Pankratz and Bostwick2005; Hor & Taylor, Reference Hor and Taylor2010), whereas others did not detect a significant difference (Black et al. Reference Black, Winokur and Nasrallah1988; Lykouras et al. Reference Lykouras, Gournellis, Fortos, Oulis and Christodoulou2000; Leadholm et al. Reference Leadholm, Rothschild, Nielsen, Bech and Ostergaard2014). Moreover, some studies reported that negative symptoms (e.g. diminished emotional expression) were inversely related to suicide risk (Fenton et al. Reference Fenton, McGlashan, Victor and Blyler1997; Bertelsen et al. Reference Bertelsen, Jeppesen, Petersen, Thorup, Ohlenschlaeger, Le Quach, Christensen, Krarup, Jorgensen and Nordentoft2007; Chang et al. Reference Chang, Chen, Hui, Chan, Lee and Chen2014); others found that negative symptoms significantly conferred risk (Havaki-Kontaxaki et al. Reference Havaki-Kontaxaki, Kontaxakis, Protopappa and Christodoulou1994; Steblaj et al. Reference Steblaj, Tavcar and Dernovsek2007), with some reporting a non-significant relationship (Nordentoft et al. Reference Nordentoft, Jeppesen, Abel, Kassow, Petersen, Thorup, Krarup, Hemmingsen and Jorgensen2002; Hawton et al. Reference Hawton, Sutton, Haw, Sinclair and Deeks2005; Jahn et al. Reference Jahn, Bennett, Park, Gur, Horan, Kring and Blanchard2016). Similar debates exist regarding positive symptoms such as delusions and hallucinations (Kelleher et al. Reference Kelleher, Connor, Clarke, Devlin, Harley and Cannon2012; Chang et al. Reference Chang, Chen, Hui, Chan, Lee and Chen2014; DeVylder et al. Reference DeVylder, Lukens, Link and Lieberman2015).

These inconsistencies suggest that determining the patterns and mechanisms of the association requires examining finer-grained predictor categories and potential methodological moderators. Past studies have often focused on different predictors related to psychosis, such as specific diagnoses (e.g. schizophrenia: Paerregaard, Reference Paerregaard1975), symptom types (Leadholm et al. Reference Leadholm, Rothschild, Nielsen, Bech and Ostergaard2014), and age of onset (Robinson et al. Reference Robinson, Harris, Harrigan, Henry, Farrelly, Prosser, Schwartz, Jackson and McGorry2010). To understand the true effect estimates of psychosis, it would be beneficial to study the effects of different categories of predictors (e.g. diagnoses, symptoms, aspects). Identifying predictor categories would also inform the field whether and how psychosis may directly and indirectly confer risk for STBs.

The mixed evidence further suggests that potential methodological differences might be impacting the effect estimates. Studies vary in their sample characteristics (e.g. sample size and age) and research design (e.g. follow-up length). Examining whether the effect of psychosis differs among populations is critical to understand the effect magnitude of psychosis.

Under these circumstances, a meta-analysis that quantitatively summarizes existing research is needed to determine the strength of psychosis as a risk factor for suicidality. As such, we conducted a meta-analysis that advances our knowledge in three major ways. First, only longitudinal studies were included in order to examine the magnitude of psychosis as a risk factor instead of a correlate. Second, we meta-analyzed categories of psychosis-related predictors (i.e. psychosis diagnosis, symptoms, and aspects) to study whether certain predictor categories are particularly strong. Third, moderator analyses were conducted to test how the effect of psychosis might vary based on sample characteristics. Understanding whether and to what extent psychosis confers risk for STBs is important for informing prediction and treatment.

Methods

Data sources, study selection, inclusion criteria

We identified all relevant articles using a range of search terms through 1 January 2016 using PubMed, PsycInfo, and Google Scholar. Search terms included combinations of words for ‘longitudinal’ and ‘suicide,’ including: ‘longitudinal,’ ‘longitudinally,’ ‘predicts,’ ‘prediction,’ ‘prospective,’ ‘prospectively,’ ‘future,’ ‘later,’ and ‘self-injury,’ ‘suicidality,’ ‘self-harm,’ ‘suicide,’ ‘suicidal behavior,’ ‘suicide attempt,’ ‘suicide death,’ ‘suicide plan,’ ‘suicide thoughts,’ ‘suicide ideation,’ ‘suicide gesture,’ ‘suicide threat,’ ‘nonsuicidal self-injury (NSSI),’ ‘self-mutilation,’ ‘deliberate self-harm (DSH),’ ‘self-cutting,’ ‘cutting,’ ‘self-burning,’ and ‘self-poisoning.’ Variants of nonsuicidal self-injury were included to increase likelihood of identifying relevant articles that may have otherwise been missed if a narrower search strategy were to have been applied. However, outcomes that combined forms of STBs and/or were not specific to STBs (e.g. parasuicide, deliberate self-harm, etc.) were excluded as we were interested in the specificity of effects on discrete suicide-relevant outcomes. We also searched the reference sections of all papers identified through these sources.

Inclusion required that studies report at least one longitudinal analysis using psychosis or related variables predicting suicide ideation, attempts, and death. We also required papers to be peer-reviewed publications in English. We elected to use only published studies for three reasons. First, we wanted to provide a summary of literature widely available to clinicians and researchers. Second, the peer review process provides some safeguards regarding study quality. Third, ensuring the completeness of unpublished data would be difficult; as such, the resulting pool of located studies could fail to be a representative sample of unpublished data, which could in turn bias results. To offset our inclusion of only published data, we have conducted extensive publication bias analyses and obtained bias-corrected effect size estimates.

Studies were excluded based on five criteria: (1) analyses were not longitudinal; (2) analyses did not examine discrete suicide-relevant outcomes (e.g. combining ideation and attempt as one outcome); (3) analyses were not reported with sufficient statistics (e.g. beta weights with no index of variance); (4) analyses used statistical tests that cannot be converted into odds or hazard ratios; and (5) analyses were conducted in the context of a primary treatment study.

A total of 2541 unique papers were identified. Based on abstracts, 719 studies were screened in. After reading the remaining articles in full, a total of 50 studies were retained (see online Supplement 1 for references of included studies). See Fig. 1 for PRISMA flowchart.

Fig. 1. PRISMA flow chart.

Study coding

Authors reviewed all eligible statistical tests in each study. Each statistical test where a psychosis-relevant variable was used to predict suicide ideation, attempt, or death was termed as an ‘effect size,’ and retained for further analysis. If a study reported effect sizes with the same predictor and same outcome over multiple time points, only the effect size from the last time point was retained, as this represented the most inclusive data point. This procedure was adopted to reduce data redundancy. Two effect sizes were excluded, resulting in 142 unique effect sizes.

The following information was extracted from each study: (1) publication year, (2) sample size, (3) sample age, (4) follow-up length, (5) predictor variables, (6) outcome variables, and (7) relevant statistics. Of note, we also extracted sample country, sample type, psychiatric medication status and diagnostic criteria of psychosis; results were highly consistent across these moderators (see online Supplement 2).

Publication year

Publication year was extracted to test the effects of time or generation.

Sample size

Number of participants at baseline assessment was extracted.

Sample age

Mean or median sample age was extracted from 79.91% of the effect sizes. All studies provided sufficient information for samples to be classified into adult, adolescent, or mixed samples of adults and adolescents.

Follow-up length

Psychosis may confer risk to different extent depending on follow-up length. Therefore, we recorded the longest follow-up interval in terms of months from each study.

Predictor variables

After extracting predictor variables for each effect size, authors organized predictor variables into three major categories: psychosis diagnosis, symptoms, and aspect. Psychosis diagnosis includes predictors such as schizophrenia and delusional disorder diagnoses. Psychosis symptoms were subcategorized into overall (e.g. total score on the Positive and Negative Syndrome Scale), positive (e.g. hallucinations), and negative symptoms (e.g. poverty of speech). Relevant symptoms not unique to psychosis (e.g. aggression) were labeled as other symptoms. Psychosis aspect includes predictors such as age of onset and duration of illness. These categories were created to understand meaningful differences between certain factors while striving to maintain a sufficient number of effect sizes in each category.

Outcome variables

For each effect size, we extracted from the article which discrete suicide outcome was examined: suicide ideation, attempt, or death (Nock, Reference Nock2010).

Relevant statistics

We extracted relevant statistics for each effect size in their original form (e.g. t tests, Cohen's d, means and standard deviations, risk ratios, chi-squared analyses, or 2 × 2 tables with rates and raw information).

Study quality

Study quality is often an issue of concern for meta-analyses due to the heterogeneity among studies. Combining studies that widely differ in their methodologies (e.g. naturalistic, experimental, single-group, case-control) might result in inaccurate effect estimates. Therefore, it is commonly recommended to assess and control for study quality. However, variations in study quality were substantially limited in this meta-analysis due to our stringent inclusion criteria: studies were required to share the same core design (i.e. longitudinal), examine common predictors (i.e. variables related to psychosis), and investigate the same three outcomes (i.e. suicide ideation, attempt, and death).

Despite sharing similar methodologies, studies can still differ in subtle ways, such as sample age, sample medication, and follow-up length. However, it is unclear how these minor differences might impact the outcomes. Therefore, we conducted moderator analyses to empirically examine the effects.

Statistical analyses

We used Comprehensive Meta Analysis Version 3.0 software (Englewood, N.J.) to conduct the meta-analysis. A random-effects model was adopted to account for heterogeneity across studies. Unlike fixed effects models, which assume that the true effect size is identical across studies, random effects models estimate within- and between-study variance to provide an estimate for the effects distribution. I 2 tests were used to measure between-study heterogeneity.

To provide a raw estimate of each effect size, we used zero-order (i.e. unadjusted) effects whenever possible (94.5%). The majority of effect sizes were either reported in terms of ORs or statistics that could be converted into ORs (n = 130). Effect sizes reported in hazard ratios (n = 12) cannot be converted into ORs and were therefore removed from main report (see online Supplement 3). Estimates greater than three standard deviations above the mean were treated as outliers (n = 2), resulting in 128 remaining effect sizes.

The overall effect of psychosis was examined across outcomes by combining all predictors. We then examined the effects of specific predictor categories for each outcome. To ensure that a reliable effect estimates, we only conducted analyses for categories with at least three effect sizes. We also conducted moderator analyses to test the impact of sample age, sample size, publication year, and study follow-up length on effect estimates. Distinct from moderator analyses in primary studies, moderator analyses in meta-analyses can only examine variations in effect size across studies instead of within each study.

Finally, we calculated multiple indices of publication bias using the following tests: Classic Fail-safe N, Orwin's Fail-safe N, Begg and Mazumdar Rank Correlation Test, Egger's Regression test, funnel plot symmetry, and Duval and Tweedie's Trim and Fill test.

Results

Descriptive statistics

The earliest published study included in this meta-analysis was published in 1975 (Paerregaard, Reference Paerregaard1975). The number of effect sizes reported has increased over the years, with 2.3% published before 1985, 6.3% between 1985 and 1994, 36.7% between 1995 and 2004, and 54.7% between 2005 and 2015. The majority of the effect sizes examined suicide death (43.0%), followed by attempt (39.1%) and ideation (18.0%). A total of 78.1% of the studies recruited adult participants; the rest recruited adolescents (10.9%), or a mix of both (10.9%). Most studies used clinical samples (68.8%), with the rest from self-injurious samples (18.0%), and community samples (13.3%). The median follow-up length was 7.5 years (M = 128.86 months, s.d. = 110.18 months). Only five effect sizes used follow-up lengths of 1 year or shorter.

Prediction estimates and publication bias

Suicide ideation

A total of 23 effect sizes were included. Overall, psychosis significantly conferred risk for suicide ideation, with a weighted mean Odds Ratio (wOR) of 1.70 [95% confidence interval (CI) 1.39–2.08]. Between-study heterogeneity was low (I 2 = 0%) and there was minimal evidence of publication bias (Table 1 and Fig. 2 a). As a predictor category, psychosis symptoms predicted higher risk for suicide ideation (Table 2). This effect was mainly driven by overall symptoms and positive symptoms; negative symptoms did not significantly predict risk.

Fig. 2. Funnel plots. (a) Suicide ideation (b) Suicide attempt (c) Suicide death. Note. Open circles represent observed estimates; shaded circles represent imputed values estimated to be missing to the left of the mean. Open diamond indicates unadjusted weighted mean hazard ratio; shaded diamond indicates adjusted weighted mean hazard ratio.

Table 1. Publication bias

OR, weighted mean odds ratio.

Notes. Classic and Orwin's Fail-safe N values represent the number of studies needed to nullify the observed effects; Begg and Mazumdar Rank Correlation Test computes the rank order correlation between effect estimates and standard error; Egger's Test of the Intercept uses precision (i.e. the inverse of the standard error) to predict the standardized effect (i.e. effect size divided by the standard error). The size of the effect is reflected in the slope and bias is reflected in the intercept (B 0); Missing effect sizes under Duval & Tweedie's Trim & Fill are the number of effect sizes estimated as missing below the mean.

Table 2. OR analyses

n, number of effect sizes; OR, weighted mean odds ratio; 95% CI, 95% confidence interval, dashes indicate unavailable information; I 2 indicates the percentage of variances due to heterogeneity between studies.

Note. Estimates were not reported for analyses involving fewer than three effect sizes as small number of effect sizes reduce the reliability of estimates.

Suicide attempt

The overall analysis included 50 effect sizes, resulting in a wOR of 1.36 (95% CI 1.25–1.48). High heterogeneity between studies (I 2 = 87.63%) and moderate publication bias (Table 1 and Fig. 2 b) were detected. Psychosis diagnoses, especially Cluster A Personality Disorders and Schizophrenia diagnoses, significantly predicted higher risk for suicide attempt. No other predictor category yielded a significant result (Table 2).

Suicide death

A total of 55 effect sizes were included in the analyses. Overall, psychosis significantly elevated risk for suicide death, with a wOR of 1.40 (95% CI 1.14–1.72). Between study heterogeneity was high (I 2 = 93.20%). Results of publication bias tests indicated minimal bias (Table 1 and Fig. 2 c). We also examined specific risk factor categories (Table 2). Psychosis diagnoses, especially unspecified diagnoses, predicted significantly higher risk for death. For psychosis symptoms, both overall symptoms and positive symptoms were associated with significantly higher risk; however, negative symptoms were associated with significantly lower risk.

Moderator analyses

Sample age

In predicting attempt, the effect estimate yielded from effect sizes using adult samples was significantly larger than those from effect sizes using adolescent and mixed samples. For suicide death, moderator analyses indicated that psychosis conferred significantly higher risk in mixed samples than in adult samples. There were no significant effects of mean/median sample age (Table 3).

Table 3. Moderator analyses

n, number of effect sizes; OR, weighted mean odds ratio; 95% CI, 95% confidence interval, dashes indicate unavailable information; b, regression coefficient.

Note. Estimates were not reported for analyses involving fewer than three effect sizes, as small number of effect sizes compromise the accuracy of estimates.

Year of publication

The effect estimates of psychosis remained consistent regardless of publication year (Table 3).

Sample size

There were no significant effects of sample size on the effect estimates of psychosis for ideation and death. Larger sample sizes were associated with larger effect estimates for attempt, though the effect was small (Table 3).

Follow-up length

Meta-regression analyses indicated that effect estimates remained statistically identical regardless of the follow-up length (Table 3).

Discussion

This meta-analysis aimed to estimate the effects of psychosis on STBs. Our major findings include: (1) psychosis significantly confers risk for suicide ideation, attempt, and death; (2) different predictor categories confer risk to various extents; (3) minor moderator effects of sample and methodological differences were detected. Each of these findings is discussed in more detail below.

Psychosis, when analyzed as an aggregate across different predictors, significantly, albeit weakly, increased risk for suicide ideation, attempt, and death. These results were consistent with previous findings showing higher prevalence rate estimates of STBs in psychotic populations (Radomsky et al. Reference Radomsky, Haas, Keshavan and Sweeney1995; Palmer et al. Reference Palmer, Pankratz and Bostwick2005; Fialko et al. Reference Fialko, Freeman, Bebbington, Kuipers, Garety, Dunn and Fowler2006). Even though mechanisms for the association between psychosis and suicidality were not directly tested in this study, our findings were consistent with certain hypotheses. The fact that the global presence and severity of psychosis (e.g. psychosis analyzed across all predictors, psychosis spectrum disorders, overall psychotic symptoms) confers risk indiscriminately across suicide outcomes suggests the likelihood that either delusions and hallucinations directly cause ideations and actions, or psychosis induces ideations and actions via other indirect mechanisms (e.g. hopelessness, perceived burdensomeness), or a combination of both.

In-depth analyses of predictor categories revealed that certain predictor categories exerted stronger effects on suicide outcomes than others. Psychosis spectrum diagnosis as a predictor category significantly predicted suicidal behaviors, indicating that psychosis might confer risk through increased capability for suicide. Schizophrenia diagnosis surprisingly did not significantly increase risk for suicide death, though we speculate that the relatively small number of studies might have rendered our analyses underpowered. The results might have also been constrained by limitations of the literature discussed in more detail below.

Further, analyses on subcategories of psychotic symptoms found distinct patterns between positive and negative symptoms as predictors. Positive symptoms significantly increased risk for both suicide ideation and death, whereas negative symptoms were not significantly associated with ideation, and in fact significantly decreased risk for death. These findings were consistent with the hypothesis that delusions and hallucinations directly induce STBs. Moreover, positive symptoms might have also rendered individuals more prone to experience painful events such as jumping from a high place in response to persecutory delusions. Therefore, it is possible that positive symptoms also confer risk indirectly through increased capability for suicide (Joiner, Reference Joiner2005; Van Orden et al. Reference Van Orden, Witte, Cukrowicz, Braithwaite, Selby and Joiner2010).

Besides positive symptoms directly inducing STBs, we speculate two other reasons that might explain the differences between positive and negative symptoms as predictors. First, the avolition and amotivation reflected in negative symptoms might have prevented individuals from making active plans for suicide attempts. Contrary to common perception of suicide as impulsive, it is suggested that suicide is often a deliberate act that involves many preparations ahead of time, such as researching lethal methods on the internet, acquiring the means, and choosing the location for suicide (Joiner, Reference Joiner2010). It is highly likely that individuals experiencing negative symptoms such as avolition and amotivation might have been unable to engage in deliberate planning for an attempt. Second, factor analytic studies have shown that negative symptoms consistently emerge as a factor separate from positive symptoms (Blanchard & Cohen, Reference Blanchard and Cohen2006). If positive and negative symptoms are distinct manifestations of psychosis, it is not surprising that they have different effects on suicidality.

Given sample and methodological variances across studies, we also conducted moderator analyses. The results showed that the effect estimates of psychosis remained largely consistent across multiple variables (e.g. sample age and type, sample size, year of publication, follow-up length), though minor moderator effects were detected for some samples.

Based on our findings, we note three major limitations of the literature and two limitations of the present study. First, certain predictors were rarely examined in a longitudinal context, which precluded us from determining the mechanisms through which psychosis confers risk for suicidality. For instance, longitudinal studies on schizophrenia diagnosis, aspects of psychosis (e.g. age of onset), and psychosis in the context of mood disorders and organic psychopathology are scarce. In a similar vein, we were unable to conduct even finer-grained analyses on specific symptoms (e.g. delusions, flat affect) due to insufficient number of effect sizes. Future research should study these less-understood factors and their longitudinal associations with suicidality.

Second, past research has often overlooked the interactions both among psychosis predictors, and between psychosis and other risk factors (e.g. demographics, comorbidity with internalizing disorders). Out of the 50 papers included in this meta-analysis, three studies reported a total of five interaction effects (Pillmann et al. Reference Pillmann, Balzuweit, Haring, Blöink and Marneros2003; Flensborg-Madsen et al. Reference Flensborg-Madsen, Knop, Mortensen, Becker, Sher and Grønbæk2009; Crocq et al. Reference Crocq, Naber, Lader, Thibaut, Drici, Everitt, Hall, Le Jeunne, Mittoux, Peuskens, Priori, Sturkenboom, Thomas, Tanghøj, Toumi, Mann and Moore2010), and these interactions were too idiosyncratic to meaningfully meta-analyze. The lack of emphasis on interactions is particularly sobering considering that accurate suicide prediction requires capitalizing on interactive effects (Ribeiro et al. Reference Ribeiro, Franklin, Fox, Bentley, Kleiman, Chang and Nock2016). According to our findings, the average odds ratios of a psychosis predictor are 1.70, 1.36, and 1.40 for suicide ideation, attempt, and death. Therefore, a psychosis risk factor on average raises one's likelihood of experiencing suicide ideation and attempt in a given year from 3% and 0.5%, to 5.1% (3% × 1.70) and 0.68% (0.5% × 1.36), and raises one's likelihood to die from suicide from 0.013% to 0.018% (0.013% × 1.40), respectively (Kessler et al. Reference Kessler, Berglund, Nock, Wang and Page2005; Kochanek et al. Reference Kochanek, Murphy, Xu and Tejada-Vera2016). These small increases in absolute risk are unlikely to be helpful in most clinical situations. Examining the interactions among multiple predictors, however, has been shown to be a promising way to greatly increase suicide prediction accuracy (Kessler et al. Reference Kessler, Warner, Ivany, Petukhova, Rose, Bromet, Brown, Cai, Colpe, Cox, Fullerton, Gilman, Gruber, Heeringa, Lewandowski-Romps, Li, Millikan-Bell, Naifeh, Nock, Rosellini, Sampson, Schoenbaum, Stein, Wessely, Zaslavsky and Ursano2015; Ribeiro et al. Reference Ribeiro, Franklin, Fox, Bentley, Kleiman, Chang and Nock2016; Walsh et al. Reference Walsh, Ribeiro and Franklin2017). Future studies should endeavor to examine interaction effects between psychosis and other factors.

Third, most studies have adopted extremely long follow-up lengths in their design. The median follow-up was 7.5 years, and only five effect sizes out of 128 used follow-up length of 1 year or shorter. Although moderator analyses indicated that results were consistent across follow-up lengths, it is possible that psychosis might have been a stronger predictor for suicide outcomes within days, weeks, or months, and that the present literature contains too few studies to reflect its short-term effect. Long follow-up lengths might have also obscured the effects of psychosis due to variations in medication status and diagnostic criteria over time. Even though moderator analyses indicated results were consistent across these two moderators, it is possible that these effects were not detected. Future research should consider using shorter follow-up lengths to approximate clinical situations and control for potential confounders.

Along with limitations in the literature, three methodological limitations should be kept in mind when interpreting the findings as well. First, the present meta-analysis only included published statistics, while excluding unpublished studies or studies without sufficient statistical information. Even though the studies included in this meta-analysis are highly homogenous (e.g. longitudinal design, psychosis predictors, suicide outcomes, similar effect sizes) and the exclusion of other studies are unlikely to affect the present findings, findings should be interpreted as a summary of peer-reviewed studies widely accessible to the public.

Second, although our findings provided evidence that was consistent or inconsistent with certain theories in the field, this meta-analysis did not directly test the mechanisms of the association between psychosis and suicide. Risk factors established through longitudinal studies precede the outcome of interest, but are still not equivalent to cause (Kraemer et al. Reference Kraemer, Kazdin, Offord, Kessler, Jensen and Kupfer1997). It is possible that a third variable (e.g. genetic antecedents, substance abuse, depression) might have caused both the risk factor and the outcome. Only experimental designs can infer cause by reducing influences of third variables through randomization and stringent control groups. However, very few experimental studies exist in the field of psychosis and suicide due to practical and ethical reasons. As such, the findings from this study reflect the longitudinal rather than causal effects of psychosis, and no definite conclusion on mechanisms can be reached based on the present meta-analysis.

Third and similarly, this study cannot rule out the possibility that potential confounders might have contributed to the association between psychosis and suicide. For example, other psychiatric disorders are common among individuals suffering from psychosis. It is estimated that 50% of patients with schizophrenia also have comorbid depression, 47% with comorbid substance use, and 29% with comorbid posttraumatic stress disorder (Buckley et al. Reference Buckley, Miller, Lehrer and Castle2009). As such, the risk that psychosis confers on suicide might be due to comorbidity with other psychiatric disorders. In addition, factors that elevate risk for both psychosis and suicide might be a potential confounder as well. For instance, research has shown impairment in neuropsychological function in suicide attempters (Keilp et al. Reference Keilp, Gorlyn, Russell, Oquendo, Burke, Harkavy-Friedman and Mann2013) and individuals with schizophrenia (Reichenberg et al. Reference Reichenberg, Harvey, Bowie, Mojtabai, Rabinowitz, Heaton and Bromet2009). Therefore, it is possible that certain neuropsychological factors may confound the association. Future studies should investigate this issue.

Together, the results of the present meta-analysis indicated that psychosis significantly confers risk for STBs, and that different predictor categories confer risk to different extents. Specifically, positive symptoms were significant risk factors, whereas negative symptoms were non-significant predictors for suicide ideation, and significant protective factors for death. Results of moderator analyses indicted some effects of sample differences, though the effects were inconsistent across outcomes. Our results provided evidence for a direct association between psychosis and suicidality through delusions and command hallucinations, and some evidence for an indirect association via hopelessness, perceived burdensomeness, and thwarted belongingness. Due to the nature of the study designs (i.e. longitudinal instead of experimental), however, the exact mechanisms of the associations between psychosis and suicide cannot be determined. Future studies should fill the gaps of research by examining less-focused risk factors, exploring interaction effects, and adopting a shorter follow-up in their design.

Supplementary material

The supplementary material for this article can be found at https://doi.org/10.1017/S0033291717002136.

Acknowledgements

None.

Declaration of Interest

None.

References

Abramson, LY, Alloy, LB, Hogan, ME, Whitehouse, WG, Gibb, BE, Hankin, BL, Cornette, MM (2000). The hopelessness theory of suicidality. In Suicide Science: Expanding the Boundaries (ed. Joiner, T. E. and Rudd, M. D.), pp. 1732. Kluwer Academic/Plenum Publishers: New York, NY.Google Scholar
Abramson, LY, Metalsky, GI, Alloy, LB (1989). Hopelessness depression: a theory-based subtype of depression. Psychological Review 96, 358372.CrossRefGoogle Scholar
Aleman, A, Hijman, R, De Haan, F, Kahn, RS (1999). Memory impairment in schizophrenia: a meta-analysis. American Journal of Psychiatry 156, 13581366.Google Scholar
Awad, AG, Voruganti, LNP (2008). The burden of schizophrenia on caregivers: a review. PharmacoEconomics 26, 149162.Google Scholar
Beck, AT (1986). Hopelessness as a predictor of eventual suicide. Annals of the New York Academy of Sciences 487, 9096.CrossRefGoogle ScholarPubMed
Bertelsen, M, Jeppesen, P, Petersen, L, Thorup, A, Ohlenschlaeger, J, Le Quach, P, Christensen, TO, Krarup, G, Jorgensen, P, Nordentoft, M (2007). Suicidal behaviour and mortality in first-episode psychosis: the OPUS trial. British Journal of Psychiatry 191, s140s146.CrossRefGoogle Scholar
Black, DW, Winokur, G, Nasrallah, A (1988). Effect of psychosis on suicide risk in 1,593 patients with unipolar and bipolar affective disorders. American Journal of Psychiatry 145, 849852.Google Scholar
Blanchard, JJ, Cohen, AS (2006). The structure of negative symptoms within schizophrenia: implications for assessment. Schizophrenia Bulletin 32, 238245.Google Scholar
Buckley, PF, Miller, BJ, Lehrer, DS, Castle, DJ (2009). Psychiatric comorbidities and schizophrenia. Schizophrenia Bulletin 35, 383402.CrossRefGoogle ScholarPubMed
Chang, WC, Chen, ESM, Hui, CLM, Chan, SKW, Lee, EHM, Chen, EYH (2014). The relationships of suicidal ideation with symptoms, neurocognitive function, and psychological factors in patients with first-episode psychosis. Schizophrenia Research 157, 1218.Google Scholar
Crocq, MA, Naber, D, Lader, MH, Thibaut, F, Drici, M, Everitt, B, Hall, GC, Le Jeunne, C, Mittoux, A, Peuskens, J, Priori, S, Sturkenboom, M, Thomas, SHL, Tanghøj, P, Toumi, M, Mann, R, Moore, ND (2010). Suicide attempts in a prospective cohort of patients with schizophrenia treated with sertindole or risperidone. Elsevier B.V. European Neuropsychopharmacology 20, 829838.CrossRefGoogle ScholarPubMed
DeVylder, JE, Lukens, EP, Link, BG, Lieberman, JA (2015). Suicidal ideation and suicide attempts among adults with psychotic experiences. JAMA Psychiatry 72, 219.CrossRefGoogle ScholarPubMed
Eisenberg, P, Lazarsfeld, PF (1938). The psychological effects of unemployment. Psychological Bulletin 35, 358390.Google Scholar
Fenton, WS, McGlashan, TH, Victor, BJ, Blyler, CR (1997). Symptoms, subtype, and suicidality in patients with schizophrenia spectrum disorders. American Journal of Psychiatry 154, 199204.Google Scholar
Fialko, L, Freeman, D, Bebbington, PE, Kuipers, E, Garety, PA, Dunn, G, Fowler, D (2006). Understanding suicidal ideation in psychosis: findings from the prevention of relapse in psychosis (PRP) trial. Acta Psychiatrica Scandinavica 114, 177186.Google Scholar
Flensborg-Madsen, T, Knop, J, Mortensen, EL, Becker, U, Sher, L, Grønbæk, M (2009). Alcohol use disorders increase the risk of completed suicide – irrespective of other psychiatric disorders. A longitudinal cohort study. Elsevier Ireland Ltd Psychiatry Research 167, 123130.Google Scholar
Folsom, D, Jeste, DV (2002). Schizophrenia in homeless persons: a systematic review of the literature 285. Acta Psychiatrica Scandinavica 105, 404413.Google Scholar
Green, MF, Kern, RS, Heaton, RK (2004). Longitudinal studies of cognition and functional outcome in schizophrenia: implications for MATRICS. Schizophrenia Research 72, 4151.CrossRefGoogle ScholarPubMed
Grunebaum, MF, Oquendo, MA, Harkavy-Friedman, JM, Ellis, SP, Li, S, Haas, GL, Malone, KM, Mann, JJ (2001). Delusions and suicidality. American Journal of Psychiatry 158, 742747.Google Scholar
Harkavy-Friedman, JM, Kimhy, D, Nelson, EA, Venarde, DF, Malaspina, D, Mann, JJ (2003). Suicide attempts in schizophrenia: the role of command auditory hallucinations for suicide. Journal of Clinical Psychiatry 64, 871874.Google Scholar
Havaki-Kontaxaki, BJ, Kontaxakis, VP, Protopappa, VA, Christodoulou, GN (1994). Suicides in a large psychiatric hospital: risk factors for schizophrenic patients. Bibliotheca Psychiatrica 165, 6371.Google Scholar
Hawton, K, Sutton, L, Haw, C, Sinclair, J, Deeks, JJ (2005). Schizophrenia and suicide: systematic review. British Journal of Psychiatry 187, 920.Google Scholar
Hor, K, Taylor, M (2010). Suicide and schizophrenia: a systematic review of rates and risk factors. Journal of Psychopharmacology 24, 8190.CrossRefGoogle ScholarPubMed
Jahn, DR, Bennett, ME, Park, SG, Gur, RE, Horan, WP, Kring, AM, Blanchard, JJ (2016). The interactive effects of negative symptoms and social role functioning on suicide ideation in individuals with schizophrenia. Schizophrenia Research 170, 271277.CrossRefGoogle ScholarPubMed
Joiner, TE (2005). Why People Die by Suicide. Harvard University Press: Cambridge, MA.Google Scholar
Joiner, TE (2010). Myths about Suicide. Harvard University Press: Cambridge, MA.Google Scholar
Kasper, E, Rogers, R, Adams, PA (1996). Dangerousness and command hallucinations: an investigation of psychotic inpatients. Bulletin of the American Academy of Psychiatry and the Law 24, 219224.Google Scholar
Keilp, JG, Gorlyn, M, Russell, M, Oquendo, MA, Burke, AK, Harkavy-Friedman, J, Mann, JJ (2013). Neuropsychological function and suicidal behavior: attention control, memory and executive dysfunction in suicide attempt. Psychological Medicine 43, 539551.Google Scholar
Kelleher, I, Connor, D, Clarke, MC, Devlin, N, Harley, M, Cannon, M (2012). Prevalence of psychotic symptoms in childhood and adolescence: a systematic review and meta-analysis of population-based studies. Psychological Medicine 42, 18571863.Google Scholar
Kessler, RC, Berglund, P, Nock, M, Wang, PS, Page, P (2005). Trends in suicide ideation, plans, gestures, and attempts in the United States, 1990–1992 to 2001–2003. Journal of the American Medical Association 293, 19901992.Google Scholar
Kessler, RC, Warner, CH, Ivany, C, Petukhova, MV, Rose, S, Bromet, EJ, Brown, M, Cai, T, Colpe, LJ, Cox, KL, Fullerton, CS, Gilman, SE, Gruber, MJ, Heeringa, SG, Lewandowski-Romps, L, Li, J, Millikan-Bell, AM, Naifeh, JA, Nock, MK, Rosellini, AJ, Sampson, NA, Schoenbaum, M, Stein, MB, Wessely, S, Zaslavsky, AM, Ursano, RJ (2015). Predicting suicides after psychiatric hospitalization in US army soldiers: the army study to assess risk and resilience in servicemembers (Army STARRS). JAMA Psychiatry 72, 4957.Google Scholar
Kim, CH, Jayathilake, K, Meltzer, HY (2003). Hopelessness, neurocognitive function, and insight in schizophrenia: relationship to suicidal behavior. Schizophrenia Research 60, 7180.CrossRefGoogle ScholarPubMed
Kochanek, KD, Murphy, SL, Xu, J, Tejada-Vera, B (2016). Deaths: Final data for 2014. National Vital Statistics Reports 65, 1122.Google Scholar
Koeda, A, Otsuka, K, Nakamura, H, Yambe, T, Fukumoto, K, Onuma, Y, Saga, Y, Yoshioka, Y, Mita, T, Mizugai, A, Sakai, A, Endo, S (2012). Characteristics of suicide attempts in patients diagnosed with schizophrenia in comparison with depression: a study of emergency room visit cases in Japan. Schizophrenia Research 142, 3139.CrossRefGoogle ScholarPubMed
Kontaxakis, V, Havaki-Kontaxaki, B, Margariti, M, Stamouli, S, Kollias, C, Christodoulou, G (2004). Suicide ideation in inpatients with acute schiozphrenia. Canadian Journal of Psychiatry-Revue Canadienne De Psychiatrie 49, 476479.Google Scholar
Kraemer, HC, Kazdin, AE, Offord, DR, Kessler, RC, Jensen, PS, Kupfer, DJ (1997). Coming to terms with the terms of risk. Archives of General Psychiatry 54, 337343.Google Scholar
Kuperberg, G, Heckers, S (2000). Schizophrenia and cognitive function. Current Opinion in Neurobiology 10, 205210.Google Scholar
Large, M, Babidge, N, Andrews, D, Storey, P, Nielssen, O (2009). Major self-mutilation in the first episode of psychosis. Schizophrenia Bulletin 35, 10121021.Google Scholar
Leadholm, AKK, Rothschild, AJ, Nielsen, J, Bech, P, Ostergaard, SD (2014). Risk factors for suicide among 34,671 patients with psychotic and non-psychotic severe depression. Elsevier Journal of Affective Disorders 156, 119125.Google Scholar
Lee, S, Lee, MTY, Chiu, MYL, Kleinman, A (2005). Experience of social stigma by people with schizophrenia in Hong Kong. British Journal of Psychiatry 186, 153157.Google Scholar
Lewine, RRJ (2005). Social class of origin, lost potential, and hopelessness in schizophrenia. Schizophrenia Research 76, 329335.Google Scholar
Liddle, PF (2000). Cognitive impairment in schizophrenia: its impact on social functioning. Acta Psychiatrica Scandinavica 101, 1116.Google Scholar
Lykouras, E, Gournellis, R, Fortos, A, Oulis, P, Christodoulou, GN (2000). Psychotic major depression in the elderly and suicide behavior. Psychiatriki 11, 177183.Google Scholar
Marwaha, S, Johnson, S (2004). Schizophrenia and employment: a review. Social Psychiatry and Psychiatric Epidemiology 39, 337349.Google Scholar
Nock, MK (2010). Self-injury. Annual Review of Clinical Psychology 6, 339363.CrossRefGoogle ScholarPubMed
Nordentoft, M, Jeppesen, P, Abel, M, Kassow, P, Petersen, L, Thorup, A, Krarup, G, Hemmingsen, R, Jorgensen, P (2002). OPUS study: suicidal behaviour, suicidal ideation and hopelessness among patients with first-episode psychosis: one-year follow-up of a randomised controlled trial. Special Issue: The European First-Episode Schizophrenia Network 181, s98s106.Google Scholar
Paerregaard, G (1975). Suicide among attempted suicides. Suicide and Life-Threatening Behavior 5, 140144.Google Scholar
Palmer, BA, Pankratz, VS, Bostwick, JM (2005). The lifetime risk of suicide in schizophrenia: a reexamination. Archives of General Psychiatry 62, 247253.Google Scholar
Papastavrou, E, Charalambous, A, Tsangari, H, Karayiannis, G (2012). The burdensome and depressive experience of caring. Cancer Nursing 35, 187194.Google Scholar
Paul, KI, Moser, K (2009). Unemployment impairs mental health: meta-analyses. Elsevier Inc. Journal of Vocational Behavior 74, 264282.Google Scholar
Pillmann, F, Balzuweit, S, Haring, A, Blöink, R, Marneros, A (2003). Suicidal behavior in acute and transient psychotic disorders. Psychiatry Research 117, 199209.Google Scholar
Radomsky, ED, Haas, GL, Keshavan, MS, Sweeney, JA (1995). Suicidal behavior in schizophrenia and other psychotic disorders. American Journal of Psychiatry 156, 15901595.Google Scholar
Reichenberg, A, Harvey, PD, Bowie, CR, Mojtabai, R, Rabinowitz, J, Heaton, RK, Bromet, E (2009). Neuropsychological function and dysfunction in schizophrenia and psychotic affective disorders. Schizophrenia Bulletin 35, 10221029.CrossRefGoogle ScholarPubMed
Ribeiro, JD, Franklin, JC, Fox, KR, Bentley, KH, Kleiman, EM, Chang, BP, Nock, MK (2016). Letter to the editor: suicide as a complex classification problem: machine learning and related techniques can advance suicide prediction – a reply to Roaldset (2016). Psychological Medicine 46, 12.Google Scholar
Robinson, J, Harris, MG, Harrigan, SM, Henry, LP, Farrelly, S, Prosser, A, Schwartz, O, Jackson, H, McGorry, PD (2010). Suicide attempt in first-episode psychosis: a 7.4 year follow-up study. Elsevier B.V. Schizophrenia Research 116, 18.Google Scholar
Shawyer, F, Mackinnon, A, Farhall, J, Sims, E, Blaney, S, Yardley, P, Daly, M, Mullen, P, Copolov, D (2008). Acting on harmful command hallucinations in psychotic disorders: an integrative approach. Journal of Nervous and Mental Disease 196, 390398.Google Scholar
Steblaj, A, Tavcar, R, Dernovsek, MZ (2007). Predictors of suicide in psychiatric hospital. Acta Psychiatrica Scandinavica 100, 383388.Google Scholar
Van Orden, KA, Witte, TK, Cukrowicz, KC, Braithwaite, SR, Selby, EA, Joiner, TE (2010). The interpersonal theory of suicide. Psychological Review 117, 575600.Google Scholar
Walsh, CG, Ribeiro, JD, Franklin, JC (2017). Predicting risk of suicide attempts over time through machine learning. Clinical Psychological Science 5, 457469.Google Scholar
Wong, Z, Öngür, D, Cohen, B, Ravichandran, C, Noam, G, Murphy, B (2013). Command hallucinations and clinical characteristics of suicidality in patients with psychotic spectrum disorders. Comprehensive Psychiatry 54, 611617.Google Scholar
Figure 0

Fig. 1. PRISMA flow chart.

Figure 1

Fig. 2. Funnel plots. (a) Suicide ideation (b) Suicide attempt (c) Suicide death. Note. Open circles represent observed estimates; shaded circles represent imputed values estimated to be missing to the left of the mean. Open diamond indicates unadjusted weighted mean hazard ratio; shaded diamond indicates adjusted weighted mean hazard ratio.

Figure 2

Table 1. Publication bias

Figure 3

Table 2. OR analyses

Figure 4

Table 3. Moderator analyses

Supplementary material: File

Huang supplementary material

Huang supplementary material 1

Download Huang supplementary material(File)
File 30.5 KB
Supplementary material: File

Huang supplementary material

Huang supplementary material 2

Download Huang supplementary material(File)
File 19.6 KB
Supplementary material: File

Huang supplementary material

Huang supplementary material 3

Download Huang supplementary material(File)
File 26.1 KB