Hostname: page-component-7b9c58cd5d-sk4tg Total loading time: 0 Render date: 2025-03-15T19:27:25.195Z Has data issue: false hasContentIssue false

A naturalistic, randomized, controlled trial combining cognitive remediation with cognitive–behavioural therapy after first-episode non-affective psychosis

Published online by Cambridge University Press:  30 October 2013

R. J. Drake*
Affiliation:
University of Manchester, Manchester Academic Health Science Centre, Manchester, UK Manchester Mental Health & Social Care NHS Trust, Manchester Academic Health Science Centre, Manchester, UK
C. J. Day
Affiliation:
Lancashire Care NHS Foundation Trust, Preston, UK
R. Picucci
Affiliation:
Lancashire Care NHS Foundation Trust, Preston, UK
J. Warburton
Affiliation:
Lancashire Care NHS Foundation Trust, Preston, UK
W. Larkin
Affiliation:
Lancashire Care NHS Foundation Trust, Preston, UK
N. Husain
Affiliation:
University of Manchester, Manchester Academic Health Science Centre, Manchester, UK Lancashire Care NHS Foundation Trust, Preston, UK
C. Reeder
Affiliation:
Institute of Psychiatry, King's College London, London, UK Oxleas NHS Foundation Trust, Dartford, Kent, UK
T. Wykes
Affiliation:
Institute of Psychiatry, King's College London, London, UK
M. Marshall
Affiliation:
University of Manchester, Manchester Academic Health Science Centre, Manchester, UK Lancashire Care NHS Foundation Trust, Preston, UK
*
*Address for correspondence: R. J. Drake, MRCPsych, PhD, Institute of Brain, Behaviour and Mental Health, University of Manchester, 3.315 Jean McFarlane Building, Oxford Road, Manchester M13 9PL, UK. (Email: richard.drake@manchester.ac.uk)
Rights & Permissions [Opens in a new window]

Abstract

Background

Cognitive remediation (CR) preceding cognitive–behavioural therapy for psychosis (CBTp) was trialled within routine clinical services, with the hypothesis that following first-episode non-affective psychosis CR would enhance CBTp efficacy by improving neuropsychological performance.

Method

A total of 61 patients with DSM-IV non-affective psychoses waiting for routine CBTp were randomized to computerized CR over 12 weeks, supported by a trained support worker, or time-matched social contact (SC). Primary outcome was the blind-rated Psychotic Symptoms Rating Scale (PSYRATS). Secondary outcomes included measures of CBTp progress, cognition, symptoms, insight and self-esteem: all at baseline, after CR (12 weeks) and after CBTp (42 weeks). PSYRATS and global neuropsychological efficacy were tested using mixed-effects models with a group × time interaction term. Measures of CBTp progress and some neuropsychological measures were modelled by regression.

Results

There was no significant difference between the CR and SC groups in PSYRATS (group × time coefficient 0.3, 95% confidence interval −0.4 to 1.1, p = 0.39). However, after CR CBTp was shorter [median 7 sessions, interquartile range (IQR) 2–12 after CR; median 13, IQR 4–18 after SC; model p = 0.011] and linked to better insight (p = 0.02). Global cognition did not improve significantly more after CR (p = 0.20) but executive function did (Wisconsin Card Sort, p = 0.012).

Conclusions

CBTp courses preceded by CR were far shorter but achieved the same outcome as CBTp preceded by an active control, consistent with neuropsychological improvement enhancing CBTp. CR was delivered by staff with minimal training, offering the potential to reduce the costs of CBTp considerably.

Type
Original Articles
Copyright
Copyright © Cambridge University Press 2013 

Introduction

Combination therapy has proven fruitful in many difficult-to-treat conditions. It has been widely touted as a means of targeting otherwise intractable symptomatic, social and cognitive outcomes of schizophrenia (Lehman et al. Reference Lehman, Kreyenbuhl, Buchanan, Dickerson, Dixon, Goldberg, Green-Paden, Tenhula, Boerescu, Tek, Sandson and Steinwachs2004; National Institute for Health and Clinical Excellence, 2009; Bird et al. Reference Bird, Premkumar, Kendall, Whittington, Mitchell and Kuipers2010). However, studies of combination therapy (especially of non-pharmacological interventions) are complex, expensive and difficult to organize. Typically, such trials have studied multiple interventions embedded within new service designs (Bertelsen et al. Reference Bertelsen, Jeppesen, Petersen, Thorup, Øhlenschlaeger, le Quach, Christensen, Krarup, Jørgensen and Nordentoft2008; Melle et al. Reference Melle, Larsen, Haahr, Friis, Johannesen, Opjordsmoen, Rund, Simonsen, Vaglum and McGlashan2008) or complex programmes of care (Hogarty et al. Reference Hogarty, Flesher, Ulrich, Carter, Greenwald, Pogue-Geile, Keshavan, Cooley, DiBarry, Garrett, Parepally and Zoretich2004; Roder et al. Reference Roder, Mueller, Mueser and Brenner2006; Guo et al. Reference Guo, Zhai, Liu, Fang, Wang, Wang, Hu, Sun, Lv, Lu, Ma, He, Guo, Xie, Wu, Xue, Chen, Twamley, Jin and Zhao2010). Whilst they have promising findings, the therapeutic offering is often so complex that it is difficult to extract precise information about the synergistic benefits, or otherwise, of combining specific interventions (Bird et al. Reference Bird, Premkumar, Kendall, Whittington, Mitchell and Kuipers2010). Studies of specific combinations of interventions are often open trials or have incompletely matched control conditions (e.g. Silverstein et al. Reference Silverstein, Hatashita-Wong, Solak, Uhlhaas, Landa, Wilkniss, Goicochea, Carpiniello, Schenkel, Savitz and Smith2005, Reference Silverstein, Spaulding, Menditto, Savitz, Liberman, Berten and Starobin2009; Greig et al. Reference Greig, Zito, Wexler, Fiszdon and Bell2007; McGurk et al. Reference McGurk, Mueser, Feldman, Wolfe and Pascaris2007; Lindenmayer et al. Reference Lindenmayer, McGurk, Mueser, Khan, Wance, Hoffman, Wolfe and Xie2008; Eack et al. Reference Eack, Greenwald, Hogarty, Cooley, DiBarry, Montrose and Keshavan2009; Bowie et al. Reference Bowie, McGurk, Mausbach, Patterson and Harvey2012). Hence, there is a pressing need for further work in this area.

Cognitive rehabilitation (CR) and cognitive–behavioural therapy for psychosis (CBTp) are good candidates for combination therapy in early non-affective psychosis for four reasons. First, both interventions have efficacy, even in the early stages of schizophrenia (Tarrier et al. Reference Tarrier, Lewis, Haddock, Bentall, Drake, Kinderman, Kingdon, Siddle, Everitt, Leadley, Benn, Grazebrook, Haley, Akhtar, Davies, Palmer and Dunn2004; Eack et al. Reference Eack, Greenwald, Hogarty, Cooley, DiBarry, Montrose and Keshavan2009). Systematic reviews show that CR improves cognitive measures across a range of domains (Wykes et al. Reference Wykes, Huddy, Cellard, McGurk and Czobor2011), effect sizes ranging from 0.15 to 0.65, whilst CBTp (Wykes et al. Reference Wykes, Steel, Everitt and Tarrier2008) improves overall symptoms, with an effect size of 0.40 [95% confidence interval (CI) 0.25–0.55]. Second, each intervention targets different but complementary aspects of the condition. Third, combining CR and CBTp makes sense in terms of service delivery, as the time spent waiting for CBTp from a specialist is a window of opportunity for computer-aided CR delivered by generic staff. Fourth, cognitive impairment is an obstacle to participation in CBTp – for example, deficits in verbal memory (logical memory test) predict reduced CBTp efficacy (Penadés et al. Reference Penadés, Catalán, Pujol, Puig, Guarch, Masana, Navarro and Gastó2010), and deficits in planning, executive function and attention are potential problems. Analogously, CR preceding social cognitive or social skills groups enhances their efficacy (Hogarty et al. Reference Hogarty, Flesher, Ulrich, Carter, Greenwald, Pogue-Geile, Keshavan, Cooley, DiBarry, Garrett, Parepally and Zoretich2004; Eack et al. Reference Eack, Greenwald, Hogarty, Cooley, DiBarry, Montrose and Keshavan2009; Bowie et al. Reference Bowie, McGurk, Mausbach, Patterson and Harvey2012), with evidence that improved global neuropsychological function and processing speed partly mediate changes in social cognitive and performance measures (e.g. Hogarty et al. Reference Hogarty, Greenwald and Eack2006). It seems logical to offer CR to people with early psychosis before they have CBTp, since improved cognitive and social functioning arising from CR should lead to improved engagement with CBTp and ultimately to better outcomes.

A unique feature of the English National Health Service (NHS) is that specialist Early Intervention Services look after all young adults with a first episode of psychosis. These services are expected to offer CBTp as part of their programme of care (Department of Health, 2001; National Institute for Health and Clinical Excellence, 2009). We planned a naturalistic trial randomizing service users to 12 weeks' CR or a parallel time-matched social contact (SC) control, both groups going on to receive CBTp as part of usual care from Early Intervention Services. CBTp offered in this way would usually last from 6 to 30 weeks, as determined by therapeutic need.

Our primary hypothesis was that preceding CBTp with CR would allow CBTp to reduce delusions and hallucinations further and earlier (i.e. enhance CBTp's efficacy). Better cognitive skills might facilitate a range of CBTp processes beyond symptom reduction, for example constructing an agreed formulation. Therefore a secondary hypothesis was that CR would allow CBTp to be completed more quickly or allow greater progress before completion (i.e. enhance CBTp's efficiency).

Method

Participants

We identified potential participants from out-patients on waiting lists for CBTp from Lancashire Care and Pennine Care Foundation NHS Trusts' Early Intervention Services. Inclusion criteria were: age 18–35 years; first episode of Diagnostic and Statistical Manual of Mental Disorders, fourth edition (DSM-IV) schizophreniform disorder, schizophrenia, schizo-affective disorder or delusional disorder, confirmed by semi-structured interview (First et al. Reference First, Spitzer, Gibbon and Williams1997). Exclusion criteria were: International Classification of Diseases (ICD)-10 organic brain disease (World Health Organization, 2010); DSM-IV substance abuse or dependence; primary diagnosis of DSM-IV substance-induced psychosis; and insufficient fluency in English to participate in neuropsychological assessment. All participants provided written informed consent and the study was approved by Bolton NHS Local Research Ethics Committee (08/H1009/76) and was consistent with the UK Research Governance Framework for Health and Social Care (Department of Health, 2005).

Outcome assessments

At baseline demographic details and medication were recorded. The Psychotic Symptom Rating Scale (PSYRATS; Haddock et al. Reference Haddock, McCarron, Tarrier and Faragher1999) was the primary outcome measure and was completed with other measures at baseline and after 12 and 42 weeks follow-up (Fig. 1). The PSYRATS validly and reliably assesses severity of delusions and hallucinations in first-episode non-affective psychosis (Drake et al. Reference Drake, Haddock, Tarrier, Bentall and Lewis2007). These are the principal targets of CBTp (Wykes et al. Reference Wykes, Steel, Everitt and Tarrier2008; National Institute for Health and Clinical Excellence, 2009) and the PSYRATS is a cognitively based measure suitable for assessing CBTp outcome. Intra-class correlation coefficients between assessors were > 0.99 for subtotals and total.

Fig. 1. Study design. CBT, Cognitive–behavioural therapy; PSYRATS, Psychotic Symptoms Rating Scale; CBTp, cognitive–behavioural therapy for psychosis.

Assessments at baseline, 12 and 42 weeks included secondary measures of symptoms and function: the Positive and Negative Symptom Scale (PANSS; Kay et al. Reference Kay, Fiszbein and Opler1987); Calgary Depression Scale for Schizophrenia (Addington et al. Reference Addington, Addington and Maticka-Tyndale1993); Rosenberg Scale for self-esteem (Rosenberg, Reference Rosenberg1965); and the Social and Occupational Functional Assessment Scale (SOFAS; Goldman et al. Reference Goldman, Skodol and Lave1992). A seven-item version of the Insight Scale (IS; Birchwood et al. Reference Birchwood, Smith, Drury, Healy, Macmillan and Slade1994), with the hospitalization item dropped for this community sample, was scored 0–14. Cronbach's α for the seven-item version was 0.71; split-half reliability was 0.56 at baseline (n = 60).

The PSYRATS was also rated at 6-week intervals during the CBTp envelope of 12–42 weeks. Participants completed these interviews by telephone if they preferred. Since CBTp was sometimes delayed in starting or prolonged, and therefore incomplete by 42 weeks, PSYRATS scales were also completed by telephone or in person at 54 weeks.

CBTp typically aims to progress from engagement, through examination of the nature of individual problems, to integration of problems into a formulation describing origins, schemata and maintaining mechanisms. At the end of CBTp therapists assessed this progression using a five-point score (Tarrier & Haddock, Reference Tarrier, Haddock, Hofmann and Thompson2001).

An assessor blind to allocation extracted number of sessions of CBTp from case records and identified readmission and relapse, defined as an exacerbation of psychotic symptoms lasting at least 2 weeks, leading to a change in management (c.f. Bebbington et al. Reference Bebbington, Craig, Garety, Fowler, Dunn, Colbert, Fornells-Ambrojo and Kuipers2006).

Neuropsychological assessments

Intelligence quotient (IQ) was assessed at baseline with the Wechsler Adult Intelligence Scale block design subtest and Wechsler Test of Adult Reading (Wechsler, Reference Wechsler1997). Secondary neuropsychological assessments of attention, executive functions and memory were completed at baseline, 12 and 42 weeks using a version of the Wisconsin Card Sort Task (WCST; Koren et al. Reference Koren, Seidman, Poyurovsky, Goldsmith, Viksman, Zichel and Klein2004), Trailmaking (Reitan, Reference Reitan1979), 0-back (Cohen et al. Reference Cohen, Forman, Braver, Casey, Servan-Schreiber and Noll1993), paragraph recall (Lezak, Reference Lezak1983; Cunje et al. Reference Cunje, Molloy, Standish and Lewis2007) and Rey–Osterrieth Complex Figure (Silverstein et al. Reference Silverstein, Osborn and Palumbo1998) tasks. Supplementary Table S1 contains details and rationales.

Allocation

Within 3 days of initial assessment assessors faxed details of participants to identified administrators independent of the trial team and unaware of the hypotheses. The first author (R.J.D., who never had contact with participants and had no further part in allocation) generated an allocation sequence before the trial using randomly permuted variable blocks (Dallal, Reference Dallal2007). The independent administrators retained the sequence and used it to allocate participants to experimental or comparison groups, masked from assessors; they then communicated allocation to the researchers providing the study interventions by telephone, before faxing them participants' details.

Interventions

CR was provided in patients' homes or service bases using the ‘Computerised Interactive Remediation of Cognition – Interactive Training for Schizophrenia’ (CIRCUITS) software (Reeder & Wykes, Reference Reeder and Wykes2010) run from a secure website via the Internet or on DVD. Based on a CR programme validated in schizophrenia patients (Wykes et al. Reference Wykes, Newton, Landau, Rice, Thompson and Frangou2007a , Reference Wykes, Reeder, Landau, Everitt, Knapp, Patel and Romeo b ), it used a colourful, engaging, virtual town as an environment guiding participants through a sequence of tasks. This provided social context for tasks, each requiring a specific mixture of skills (e.g. sustained attention, working memory, registration and recall, planning) and with specific criteria for progression. Early tasks prepared for later ones of increasing difficulty and complexity. Trainers supported participants and could enter their virtual environment with privileges that allowed them to modify the sequence.

Trainers were psychology graduates with 1 week's specific training to deliver CIRCUITS (on a regular, open course provided by C.R. and T.W.), which is based on well-established CR principles (see Table 1 and online Supplementary Tables S1 and S2). They were trained to offer encouragement, alter task difficulty and performance, and sessions' length and scheduling to enable participants to progress and remain engaged.

Table 1. Critical features of CIRCUITS cognitive remediation (Wykes & Reeder, Reference Wykes and Reeder2005)

CIRCUITS, Computerised Interactive Remediation of Cognition – Interactive Training for Schizophrenia.

The comparison condition was SC with support workers, the duration of exposure matching exposure to CR trainers. Both conditions provided interpersonal contact, warmth and unconditional positive regard within a professional relationship. Social activity (conversation, neurocognitively undemanding recreations) was the basis of sessions, though where necessary workers supported participants sufficiently to maintain their motivation to attend (i.e. non-directive listening concerning problems and symptoms).

CBTp was provided separately by therapists employed specifically for that purpose by the two NHS services, working from seven different community clinics. All had specific CBTp training and were supervised by experienced senior CBTp therapists according to NICE standards (National Institute for Health and Clinical Excellence, 2009). CBTp quality was assessed using the Cognitive Therapy Scale for psychosis (CTSpsy; Haddock et al. Reference Haddock, Devane, Bradshaw, McGovern, Tarrier, Kinderman, Baguley, Lancashire and Harris2001), independently rated from randomly selected audiotaped sessions provided that therapists thought it clinically appropriate and patients provided separate written consent. The mean CTSpsy score for seven CBTp interviews was 75.2% (s.d. = 14.5%).

Analysis

The effect of group on PSYRATS score was estimated using a mixed-effects model estimated using full information maximum likelihood with Stata v. 11.2 (StataCorp LP, USA). Allocation group and a group × time interaction term modelled the intervention's effect. Time was entered as square root of weeks from baseline: since PSYRATS scores tended to level off during follow-up this improved model fit (Rabe-Hesketh & Skrondal, Reference Rabe-Hesketh and Skrondal2005). Demographic and other baseline potential confounders (lack of antipsychotic prescription, previous illicit substance use, diagnosis of schizophrenia) were removed by backward elimination, the criterion for retention being p < 0.20. PANSS, SOFAS, depression and self-esteem scales were analysed in the same way. Insight, in its mixed-effects model, was transformed towards normality by squaring IS score. Insight changed relatively linearly over time, so ‘weeks from baseline’ was entered as the time variable. A logistic regression against drop-out with baseline variables as predictors assessed the pattern of missing data in relation to these variables.

As a categorical outcome, remission was defined as a PSYRATS score of zero. Groups were compared using Fisher's exact test.

Number of sessions was compared between allocation groups by the Mann–Whitney U test, and adjusted for potential confounders by entering allocation and baseline variables into a backward linear regression (removal threshold p > 0.20), clustered by unit.

For neuropsychological measures, a global score was calculated. Individual baseline scores were transformed to normal distributions with a mean of 100 (s.d. = 15) and higher scores indicating better performance before deriving a baseline global score with the same characteristics. The same transformations were applied to 12- and 42-week follow-up scores, giving distributions with various means and standard deviations that demonstrated changes in global cognition. Global score was then entered into a mixed-effects model.

In this relatively able population WCST categories complete and complex figure copying (rather than recall: online Supplementary Table S3) scores were too skewed by ceiling effects for this process. Categories complete and copying were modelled by ordinal logistic regression against follow-up scores, adjusting for baseline scores and potential confounders after backward elimination, clustering by clinic. To examine the sensitivity of ordinal logistic and linear regressions to the effect of drop-out, all were repeated with case probability weighted by a function of the risk of attrition, derived from the logistic regression against drop-out.

Time to relapse and readmission for allocation groups were compared by log-rank tests. Cox regressions adjusted hazard ratios (HRs) for potential demographic and clinical confounders, stratified by clinic.

Sample size

Assuming, after the trial by Eack et al. (Reference Eack, Greenwald, Hogarty, Cooley, DiBarry, Montrose and Keshavan2009), a PSYRATS effect size of 0.5, recruiting 64 participants provides >80% power with two-tailed α of 0.05, correlation of 0.5 between seven successive assessments, and 25% drop-out (Stata 11.2; StataCorp LP, USA).

Results

Participant characteristics

A total of 272 CBTp waiting-list patients were screened (Fig. 2). A total of 66 consented to participate but after four withdrew, 62 were randomized. One was excluded afterwards since a concealed history of previous psychosis rendered him ineligible, so 61 entered a modified intention-to-treat analysis (Table 2). In all, seven in the CR group and four in the SC group were prescribed no maintenance antipsychotic (p = 0.51, Fisher's exact test): one in the CR and two in the SC group first-generation antipsychotics; and the rest second-generation antipsychotics. No anticholinergics were prescribed and only one in each group took medication with high muscarinic receptor affinity (clozapine). In total, 34 were recruited from Lancashire Care's three units and 27 from Pennine Care's four units, without significant differences between units in allocation group (p = 0.24, Cramér's V) or baseline PSYRATS (p = 0.21, Kruskal–Wallis test).

Fig. 2. Consolidated Standards of Reporting Trials (CONSORT) flow diagram for trial participants. SC, Social contact; CR, cognitive remediation; CBTp, cognitive–behavioural therapy for psychosis.

Table 2. Group demographic and clinical characteristics

s.d., Standard deviation; IQR, interquartile range; PSYRATS, Psychotic Symptoms Rating Scale; PANSS, Positive and Negative Symptom Scale; IQ, intelligence quotient; WCST, Wisconsin Card Sort Task; WHO, Word Health Organization.

a WHO (2012).

A total of 10 (16%) withdrew before or during CR or SC. Also, 17 (28%) did not complete the final follow-ups, of whom four (7%) provided 12-week data. Only 12 (20%) withdrew and provided no follow-up data (another one provided only CBTp data).

Assessors accidentally discovered two participants' allocation group during the trial, one in each group. CBTp therapists discovered eight participants' allocation group (five CR and three SC) but their guesses at allocation of the other participants were no better than chance (p = 0.43, Fisher's exact test). Logistic regression against completion found no significant predictors of drop-out.

There was no significant difference in exposure to CR trainers or support workers in the SC condition: median 390 min [interquartile range (IQR) 145–610 min] for SC, 375 min (IQR 125–600 min) for CR (p = 0.84, Mann–Whitney U test); median 8.5 (IQR 3–10) sessions of SC, 9 (IQR 4–13) sessions of CR (p = 0.20, Mann–Whitney U test). Mean length of sessions was 48.3 min (s.d. = 15.2 min) for SC and 43.9 min (s.d. = 17.4 min) for CR (p = 0.39, t test). Most therapist sessions were negotiated to occur weekly, with ‘homework’ in between for CR.

Did CR improve cognition?

We explored whether CR predicted differences in global cognition scores. It did not: intercept (baseline difference between allocation groups) −1.7 (95% CI −7.7 to 4.4, p = 0.59); group × time −0.73 (95% CI −1.84 to 0.38, p = 0.20) (see online Supplementary Table S4). However, after intervention the CR group completed significantly more WCST categories: after CR, 74% completed five categories (range 4–5); after SC, 63% (range 2–5); adjusted odds ratio (OR) 2.9 (95% CI 1.3–6.9, p = 0.012). By final follow-up these gains were lost (adjusted OR 0.7, 95% CI 0.2–2.5, p = 0.61). Median score for complex figure copying was 35 (IQR 33–36) after CR, against 33 (IQR 29–35) after SC (p = 0.11, Mann–Whitney U test). An outlier with persistent, specific, severe visuospatial impairment was removed: baseline copying score 12; range of other participants' scores 24–36, mean 32.89, s.d. 3.89 (Dixon's Q 0.50, p < 0.01; Rorabacher, Reference Rorabacher1991). CR had a trend towards better scores at 12 weeks (adjusted OR 3.83, 95% CI 0.99–14.77, p = 0.052) but not at the final assessment (adjusted OR 1.6, 95% CI 0.3–7.7, p = 0.56).

Did CR improve CBTp's efficacy?

Symptoms

CR was not associated with significantly lower PSYRATS scores over the period of study (Fig. 3): intercept (baseline group difference) was −2.8 (95% CI −10.7 to 5.2, p = 0.50); group × time term was 0.3 (95% CI −0.4 to 1.1, p = 0.39) (see online Supplementary Table S5). The effect of CR on PANSS was non-significant (intercept 3.8, 95% CI −2.7 to 10.3, p = 0.25; group × time −0.4, 95% CI −1.5 to 0.6, p = 0.44). Eight participants in each allocation group had full remission of symptoms on PSYRATS (OR 0.96, 95% CI 0.26–3.51, Fisher's exact test p = 1.00).

Fig. 3. Mean auditory hallucination (AH) and Delusion subscale scores by visit for each allocation group. PSYRATS, Psychotic Symptoms Rating Scale.

However, over follow-up insight changed significantly more positively after CR than SC (baseline −1.9, 95% CI −27.7 to 23.8, p = 0.88; group × time 0.55, 95% CI 0.08 to 1.01, p = 0.02). At 42 weeks median insight score was 12 after both CR and SC, but IQRs after CBTp (9.5–13 for CR, 6–13 for SC) showed that fewer of the lowest scorers (e.g. scores under half maximum, 7/14) were in the CR group (see online Supplementary Fig. S1). There was no significant effect of CR on SOFAS, depression or self-esteem.

Repeating multiple regressions for clinical and neuropsychological variables, weighting cases according to the probability of drop-out did not alter the significance of results except for complex figure copying. The 12-week complex figure copying, previously showing trend-level differences in scores, became significantly improved after CR (adjusted OR 5.0, 95% CI 1.3, 20.0, p = 0.019).

Relapse and readmission

CR had no significant effect on time to relapse (adjusted HR 1.4, 95% CI 0.5–3.5, p = 0.50) or readmission (adjusted HR 1.2, 95% CI 0.2–5.5, p = 0.84).

Did CR improve CBTp's efficiency?

CBTp after CR required far fewer sessions: median 7 (IQR 2–12), compared with 13 (IQR 6–18) for the SC group (p = 0.039, Mann–Whitney U test) (see online Supplementary Fig. S2). CR still predicted fewer sessions after adjustment for potential confounders (coefficient −1.0, 95% CI −0.3 to −1.7, p = 0.011). Unmasking was not responsible: 53 participants' allocations remained masked from therapists; their median was 6 sessions (IQR 1–12) after CR and 15 (IQR 7–19) after SC (p = 0.012, Mann–Whitney U test). Eight participants' allocation was unmasked to therapists: after CR, their median number of sessions was 14 (IQR 10–18.5); against 9 (IQR 3–18) after SC (p = 0.37, Mann–Whitney U test). There was no evidence that level of formulation differed between CR and SC across all participants (adjusted OR 1.1, 95% CI 0.2–6.1, p = 0.88) or in masked patients alone (adjusted OR 0.7, 95% CI 0.2–2.3, p = 0.50).

Discussion

Although the hypothesis that CR would enhance CBTp's effect on symptoms was rejected, the hypothesis that it would improve CBTp's efficiency was supported. After CR, participants made the same progress in half the CBTp sessions. If CBTp ended when therapists judged that clients had progressed cognitively as far as possible, then better engagement due to improved insight and cognition after CR might have helped client and therapist reach this point more quickly. If symptom reduction determined when therapists ended CBTp, less efficient CBTp after SC could have led therapists to lengthen CBTp to compensate. Either would explain why efficiency rather than efficacy improved.

The CR group had fewer low scores on the IS at 42 weeks. The difference from the SC group became substantially greater after CBTp than before, so it appears most likely that CR enhanced CBTp and this led to relatively better insight after CBTp than in the comparison group. CR improved WCST performance, though this was not sustained. The WCST is one of the most common, robust and sensitive measures of CR outcome (Wykes et al. Reference Wykes, Huddy, Cellard, McGurk and Czobor2011) and frequently predicts progress in social skills and CR interventions (Kurtz, Reference Kurtz2011), particularly categories complete (schema generation; Reeder et al. Reference Reeder, Smedley, Butt, Bogner and Wykes2006). Any simultaneous improvement in complex figure copying could also be explained by improved executive control (Silverstein et al. Reference Silverstein, Osborn and Palumbo1998). Silverstein et al. (Reference Silverstein, Hatashita-Wong, Solak, Uhlhaas, Landa, Wilkniss, Goicochea, Carpiniello, Schenkel, Savitz and Smith2005) previously proposed that transient cognitive benefits could promote sustained changes in behaviour and performance during succeeding interventions, consistent with temporary neuropsychological gains facilitating CBTp.

Global cognition did not improve but other neuropsychological measures might have been less accurate than the WCST. Short, less demanding cognitive tasks were chosen for these process measures, mindful of the risk of alienating potential and actual participants and rendering the sample unrepresentative. Though these tasks had demonstrated validity in other trials (Supplementary Table S1) they lacked the rigor of lengthier cognitive batteries (e.g. MATRICS Consensus Cognitive Battery; Nuechterlein et al. Reference Nuechterlein, Green, Kern, Baade, Barch, Cohen, Essock, Fenton, Frese, Gold, Goldberg, Heaton, Keefe, Kraemer, Mesholam-Gately, Seidman, Stover, Weinberger, Young, Zalcman and Marder2008), making them vulnerable to practice effects that could have concealed CR's benefits. Nevertheless, other CR trials with more extensive batteries found that executive function effect sizes exceeded other types of cognition (Wykes et al. Reference Wykes, Reeder, Landau, Everitt, Knapp, Patel and Romeo2007b ; Dickinson et al. Reference Dickinson, Tenhula, Morris, Brown, Peer, Spencer, Li, Gold and Bellack2009), suggesting the difference was real.

Of the participants, 20% provided no data after baseline and 28% dropped out before final follow-up, in line with the sample size calculation. There was no indication that data were other than ‘missing at random’, which does not threaten the assumptions of mixed-effects models (Rabe-Hesketh & Skrondal, Reference Rabe-Hesketh and Skrondal2005); sensitivity analyses indicated little effect of bias on the results. Data on those who completed CBTp was evenly distributed by allocation, also arguing against selection bias influencing results, but attrition remains a limitation.

Another limitation of our study was that inexperienced therapists delivered CR. CBTp was also delivered by NHS clinicians who, though qualified and supervised, were not research trained and supervised (although CBTp quality scores were reasonably good). Even so, we felt it important to accept these limitations in order to simulate routine clinical practice and make generalization credible. The naturalistic trial design had an additional strength in that NHS Early Intervention Services take nearly all incident schizophrenias in their catchment areas, so attendees at these services are reasonably representative of all those who might benefit from CBTp.

A further strength was that contact time was well matched between CR and SC groups, ensuring plausible matching of non-specific benefits of remediation. Even the most carefully designed randomized trials of combination therapy, for instance cognitive remediation with social skills training (Silverstein et al. Reference Silverstein, Hatashita-Wong, Solak, Uhlhaas, Landa, Wilkniss, Goicochea, Carpiniello, Schenkel, Savitz and Smith2005, Reference Silverstein, Spaulding, Menditto, Savitz, Liberman, Berten and Starobin2009; Bowie et al. Reference Bowie, McGurk, Mausbach, Patterson and Harvey2012) or with vocational interventions (Greig et al. Reference Greig, Zito, Wexler, Fiszdon and Bell2007; McGurk et al. Reference McGurk, Mueser, Feldman, Wolfe and Pascaris2007; Lindenmayer et al. Reference Lindenmayer, McGurk, Mueser, Khan, Wance, Hoffman, Wolfe and Xie2008), struggled to match exposure to intervention and comparison conditions. However, SC offered more opportunity for non-directive listening than task-oriented CR. If so, this compensatory therapeutic effect could have led us to underestimate the specific benefits of CR.

Allocation was almost completely masked from assessors, whereas most trials combining psychosocial interventions have been open (see above, also Hogarty et al. Reference Hogarty, Flesher, Ulrich, Carter, Greenwald, Pogue-Geile, Keshavan, Cooley, DiBarry, Garrett, Parepally and Zoretich2004; Bertelsen et al. Reference Bertelsen, Jeppesen, Petersen, Thorup, Øhlenschlaeger, le Quach, Christensen, Krarup, Jørgensen and Nordentoft2008; Melle et al. Reference Melle, Larsen, Haahr, Friis, Johannesen, Opjordsmoen, Rund, Simonsen, Vaglum and McGlashan2008; Guo et al. Reference Guo, Zhai, Liu, Fang, Wang, Wang, Hu, Sun, Lv, Lu, Ma, He, Guo, Xie, Wu, Xue, Chen, Twamley, Jin and Zhao2010). While it is unclear how far measurement of neuropsychological performance or employment rates are affected by open rating, previous meta-analysis of CBTp suggests that symptoms are sensitive, since limiting inclusion to studies with masking and other rigorous design features diminished CBTp's effect size from 0.40 to 0.22 (Wykes et al. Reference Wykes, Steel, Everitt and Tarrier2008). Though it was difficult to mask allocation from CBTp therapists, they identified allocation in only 13% of participants. In fact, unmasking only attenuated the difference in CBTp length between CR and SC groups: the reduction in CBTp length after CR was greater in those participants with allocation still masked.

Overall, our findings suggest that CR delivered by relatively unskilled workers improved the efficiency of subsequent CBTp, enabling participants and therapists to achieve the same progress in therapy in approximately half the number of sessions. The computer-aided CR approach that we selected was delivered by relatively inexpensive support workers within a typical clinical service. Labour costs for a course of CR were only £85 ($129). A substantial increase in CBTp's efficiency implies that the same number of CBTp therapists could treat many more patients; while estimates from previous studies (Kuipers et al. Reference Kuipers, Fowler, Garety, Chisholm, Freeman, Dunn, Bebbington and Hadley1998; Startup et al. Reference Startup, Jackson, Evans and Bendix2005) imply that the shortened duration was equivalent to savings of £335 ($508) per patient.

Supplementary material

For supplementary material accompanying this paper visit http://dx.doi.org/10.1017/S0033291713002559.

Acknowledgements

The authors acknowledge statistical advice from Dr Richard Emsley, University of Manchester Institute of Population Health, and the rest of the team working on the National Institute for Health Research (NIHR) HeLPER programme, particularly Ms Lauren Hampson. T.W. would like to acknowledge the support of the NIHR Biomedical Research Centre in Mental Health at the South London and Maudsley NHS Foundation Trust and an NIHR Senior Investigator Award.

This report presents independent research commissioned by the NIHR under The HeLPER Programme Grant (RP PG-0606-1302). The views expressed in this paper are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. The ISRCTN reference is 171 606 73 and the UK Clinical Research Network (UKCRN) reference is 4755.

Declaration of Interest

None.

References

Addington, D, Addington, J, Maticka-Tyndale, E (1993). Assessing depression in schizophrenia: the Calgary Depression Scale. British Journal of Psychiatry (Suppl. 22), S39S44.CrossRefGoogle ScholarPubMed
Bebbington, PE, Craig, T, Garety, P, Fowler, D, Dunn, G, Colbert, S, Fornells-Ambrojo, M, Kuipers, E (2006). Remission and relapse in psychosis: operational definitions based on case-note data. Psychological Medicine 36, 15511562.CrossRefGoogle ScholarPubMed
Bertelsen, M, Jeppesen, P, Petersen, L, Thorup, A, Øhlenschlaeger, J, le Quach, P, Christensen, , Krarup, G, Jørgensen, P, Nordentoft, M (2008). Five-year follow-up of a randomized multicenter trial of intensive early intervention vs standard treatment for patients with a first episode of psychotic illness: the OPUS trial. Archives of General Psychiatry 65, 762771.Google Scholar
Birchwood, M, Smith, J, Drury, V, Healy, J, Macmillan, F, Slade, M (1994). A self-report Insight Scale for psychosis: reliability, validity and sensitivity to change. Acta Psychiatrica Scandinavica 89, 6267.Google Scholar
Bird, V, Premkumar, P, Kendall, T, Whittington, C, Mitchell, J, Kuipers, E (2010). Early intervention services, cognitive–behavioural therapy and family intervention in early psychosis: systematic review. British Journal of Psychiatry 197, 350356.Google Scholar
Bowie, CR, McGurk, SR, Mausbach, B, Patterson, TL, Harvey, PD (2012). Combined cognitive remediation and functional skills training for schizophrenia: effects on cognition, functional competence, and real-world behavior. American Journal of Psychiatry 169, 710718.CrossRefGoogle ScholarPubMed
Cohen, JD, Forman, SD, Braver, TS, Casey, BJ, Servan-Schreiber, D, Noll, DC (1993). Activation of the prefrontal cortex in a nonspatial working memory task with functional MRI. Human Brain Mapping 1, 293304.Google Scholar
Cunje, A, Molloy, DW, Standish, TI, Lewis, DL (2007). Alternate forms of logical memory and verbal fluency tasks for repeated testing in early cognitive changes. International Psychogeriatrics 19, 6575.Google Scholar
Dallal, GE (2007). First generator, algorithm displayed from 3 August 2007 (http://www.randomization.com/). Accessed 20 January 2009.Google Scholar
Department of Health (2001). The Mental Health Policy Implementation Guide. Department of Health: London.Google Scholar
Department of Health (2005). Research Governance Framework for Health and Social Care, 2nd edn. Department of Health: London.Google Scholar
Dickinson, D, Tenhula, W, Morris, S, Brown, C, Peer, J, Spencer, K, Li, L, Gold, JM, Bellack, AS (2009). A randomized, controlled trial of computer-assisted cognitive remediation for schizophrenia. American Journal of Psychiatry 17, 170180.Google Scholar
Drake, R, Haddock, G, Tarrier, N, Bentall, R, Lewis, S (2007). The Psychotic Symptom Rating Scales (PSYRATS): their usefulness and properties in first episode psychosis. Schizophrenia Research 89, 119122.Google Scholar
Eack, SM, Greenwald, DP, Hogarty, SS, Cooley, SJ, DiBarry, AL, Montrose, DM, Keshavan, MS (2009). Cognitive enhancement therapy for early-course schizophrenia: effects of a two-year randomized controlled trial. Psychiatric Services. 60, 14681476.Google Scholar
First, MB, Spitzer, RL, Gibbon, M, Williams, JB (1997). Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I), Clinician Version, User's Guide. Arlington, VA: American Psychiatric Press.Google Scholar
Goldman, HH, Skodol, AE, Lave, TR (1992). Revising axis V for DSM-IV: a review of measures of social functioning. American Journal of Psychiatry 149, 11481156.Google ScholarPubMed
Greig, TC, Zito, W, Wexler, BE, Fiszdon, J, Bell, M (2007). Improved cognitive function in schizophrenia after one year of cognitive training and vocational services. Schizophrenia Research 96, 156161.Google Scholar
Guo, X, Zhai, J, Liu, Z, Fang, M, Wang, B, Wang, C, Hu, B, Sun, X, Lv, L, Lu, Z, Ma, C, He, X, Guo, T, Xie, S, Wu, R, Xue, Z, Chen, J, Twamley, EW, Jin, H, Zhao, J (2010). Effect of antipsychotic medication alone vs combined with psychosocial intervention on outcomes of early-stage schizophrenia: a randomized, 1-year study. Archives of General Psychiatry 67, 895904.Google Scholar
Haddock, G, Devane, S, Bradshaw, T, McGovern, J, Tarrier, N, Kinderman, P, Baguley, I, Lancashire, S, Harris, N (2001). An investigation into the psychometric properties of the Cognitive Therapy Scale for Psychosis (CTSPsy). Behavioural and Cognitive Psychotherapy 29, 221233.Google Scholar
Haddock, G, McCarron, J, Tarrier, N, Faragher, EB (1999). Scales to measure dimensions of hallucinations and delusions: the Psychotic Symptom Rating Scales (PSYRATS). Psychological Medicine 29, 879889.Google Scholar
Hogarty, GE, Flesher, S, Ulrich, R, Carter, M, Greenwald, D, Pogue-Geile, M, Keshavan, M, Cooley, S, DiBarry, AL, Garrett, A, Parepally, H, Zoretich, R (2004). Cognitive enhancement therapy for schizophrenia: effects of a 2-year randomized trial on cognition and behavior. Archives of General Psychiatry 61, 866876.CrossRefGoogle ScholarPubMed
Hogarty, GE, Greenwald, D, Eack, S (2006). Durability and mechanism of effects of cognitive enhancement therapy. Psychiatric Services 57, 17511757.Google Scholar
Kay, SR, Fiszbein, A, Opler, LA (1987). The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophrenia Bulletin 13, 261276.Google Scholar
Koren, D, Seidman, LJ, Poyurovsky, M, Goldsmith, M, Viksman, P, Zichel, S, Klein, E (2004). The neuropsychological basis of insight in first-episode schizophrenia: a pilot metacognitive study. Schizophrenia Research 70, 195202.CrossRefGoogle ScholarPubMed
Kuipers, E, Fowler, D, Garety, P, Chisholm, D, Freeman, D, Dunn, G, Bebbington, P, Hadley, C (1998). London–East Anglia randomised controlled trial of cognitive–behavioural therapy for psychosis. III: Follow-up and economic evaluation at 18 months. British Journal of Psychiatry 173, 6168.CrossRefGoogle ScholarPubMed
Kurtz, MM (2011). Neurocognition as a predictor of response to evidence-based psychosocial interventions in schizophrenia: what is the state of the evidence? Clinical Psychology Review 31, 663672.Google Scholar
Lehman, AF, Kreyenbuhl, J, Buchanan, RW, Dickerson, FB, Dixon, LB, Goldberg, R, Green-Paden, LD, Tenhula, WN, Boerescu, D, Tek, C, Sandson, N, Steinwachs, DM (2004). The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003. Schizophrenia Bulletin 30, 193217.Google Scholar
Lezak, M (1983). Neuropsychological Assessment, 2nd edn., pp. 395402. Oxford University Press: New York.Google Scholar
Lindenmayer, JP, McGurk, SR, Mueser, KT, Khan, A, Wance, D, Hoffman, L, Wolfe, R, Xie, H (2008). A randomized controlled trial of cognitive remediation among inpatients with persistent mental illness. Psychiatric Services 59, 241247.CrossRefGoogle ScholarPubMed
McGurk, SR, Mueser, KT, Feldman, K, Wolfe, R, Pascaris, A (2007). Cognitive training for supported employment: 2–3 year outcomes of a randomized controlled trial. American Journal of Psychiatry 164, 437441.CrossRefGoogle ScholarPubMed
Melle, I, Larsen, TK, Haahr, U, Friis, S, Johannesen, JO, Opjordsmoen, S, Rund, BR, Simonsen, E, Vaglum, P, McGlashan, T (2008). Prevention of negative symptom psychopathologies in first-episode schizophrenia: two-year effects of reducing the duration of untreated psychosis. Archives of General Psychiatry 65, 634640.Google Scholar
National Institute for Health and Clinical Excellence (2009). Schizophrenia Core Interventions in the Treatment and Management of Schizophrenia in Adults in Primary and Secondary Care. NICE Clinical Guideline 82. National Institute for Health and Clinical Excellence: Manchester.Google Scholar
Nuechterlein, KH, Green, MF, Kern, RS, Baade, LE, Barch, DM, Cohen, JD, Essock, S, Fenton, WS, Frese, FJ, Gold, JM, Goldberg, T, Heaton, RK, Keefe, RS, Kraemer, H, Mesholam-Gately, R, Seidman, LJ, Stover, E, Weinberger, DR, Young, AS, Zalcman, S, Marder, SR (2008). The MATRICS Consensus Cognitive Battery: part 1. Test selection, reliability, and validity. American Journal of Psychiatry 165, 203213.Google Scholar
Penadés, R, Catalán, R, Pujol, N, Puig, O, Guarch, J, Masana, G, Navarro, V, Gastó, C (2010). Is memory impairment a rate limiter in cognitive behavioural therapy for chronic schizophrenia? Psychotherapy and Psychosomatics 79, 129130.Google Scholar
Rabe-Hesketh, S, Skrondal, A (2005). Multilevel and Longitudinal Modelling using Stata. Stata Press: College Station, TX.Google Scholar
Reeder, C, Smedley, N, Butt, K, Bogner, D, Wykes, T (2006). Cognitive predictors of social functioning improvements following cognitive remediation for schizophrenia. Schizophrenia Bulletin 32 (Suppl. 1), S123S131.Google Scholar
Reeder, C, Wykes, T (2010). Computerised Interactive Remediation of Cognition – Interactive Training for Schizophrenia (CIRCUITS). King's College London: London.Google Scholar
Reitan, RM (1979). Manual for the Administration of Neuropsychological Test Batteries for Adults and Children. Ralph M. Reitan: Tucson, AZ.Google Scholar
Roder, V, Mueller, DR, Mueser, KT, Brenner, HD (2006). Integrated psychological therapy (IPT) for schizophrenia: is it effective? Schizophrenia Bulletin 32 (Suppl. 1), S81S93.Google Scholar
Rorabacher, DB (1991). Statistical treatment for rejection of deviant values: critical values of Dixon's “Q” parameter and related subrange ratios at the 95% confidence level. Analytical Chemistry 63, 139146.Google Scholar
Rosenberg, M (1965) Society and the Adolescent Self-Image. Princeton University Press: Princeton, NJ.Google Scholar
Silverstein, SM, Hatashita-Wong, M, Solak, EA, Uhlhaas, P, Landa, Y, Wilkniss, SM, Goicochea, C, Carpiniello, K, Schenkel, LS, Savitz, A, Smith, TE (2005). Effectiveness of a two-phase cognitive rehabilitation intervention for severely impaired schizophrenia patients. Psychological Medicine 35, 829837.CrossRefGoogle ScholarPubMed
Silverstein, SM, Osborn, LM, Palumbo, DR (1998). Rey–Osterrieth Complex Figure. Test performance in acute, chronic, and remitted schizophrenia patients. Journal of Clinical Psychology 54, 985994.Google Scholar
Silverstein, SM, Spaulding, WD, Menditto, AA, Savitz, A, Liberman, RP, Berten, S, Starobin, H (2009). Attention shaping: a reward-based learning method to enhance skills training outcomes in schizophrenia. Schizophrenia Bulletin 35, 222232.CrossRefGoogle ScholarPubMed
Startup, M, Jackson, MC, Evans, KE, Bendix, S (2005). North Wales randomized controlled trial of cognitive behaviour therapy for acute schizophrenia spectrum disorders: two-year follow-up and economic evaluation. Psychological Medicine 35, 13071316.Google Scholar
Tarrier, N, Haddock, G (2001). Cognitive–behavioural therapy for schizophrenia: a case formulation approach. In Treating Chronic and Severe Mental Disorders: A Handbook of Empirically Supported Interventions (ed. Hofmann, S. G. and Thompson, M. C.), pp. 6997. Guilford: New York.Google Scholar
Tarrier, N, Lewis, SW, Haddock, G, Bentall, R, Drake, R, Kinderman, P, Kingdon, D, Siddle, R, Everitt, J, Leadley, K, Benn, A, Grazebrook, K, Haley, C, Akhtar, S, Davies, L, Palmer, S, Dunn, G (2004). 18 month follow-up of a randomised, controlled trial of cognitive–behaviour therapy in first episode and early schizophrenia. British Journal of Psychiatry 184, 231239.Google Scholar
Wechsler, D (1997) WMS-III Manual: Wechsler Memory Scale, third edn. Psychological Corporation: San Antonio.Google Scholar
World Health Organization (2010). International Statistical Classification of Diseases and Related Health Problems 10th Revision (ICD-10) Version for 2010 (http://apps.who.int/classifications/icd10/browse/2010/en#/VI). Accessed 2 December 2012.Google Scholar
World Health Organization (2012). ACT/DDD index (http://www.whocc.no/atc_ddd_index/?code=N05A). Accessed 27 October 2012.Google Scholar
Wykes, T, Huddy, V, Cellard, C, McGurk, SR, Czobor, P (2011). A meta-analysis of cognitive remediation for schizophrenia: methodology and effect sizes. American Journal of Psychiatry 168, 472485.Google Scholar
Wykes, T, Newton, E, Landau, S, Rice, C, Thompson, N, Frangou, S (2007 a). Cognitive remediation therapy (CRT) for young early onset patients with schizophrenia: an exploratory randomized controlled trial. Schizophrenia Research 94, 221230.Google Scholar
Wykes, T, Reeder, C, Landau, S, Everitt, B, Knapp, M, Patel, A, Romeo, R (2007 b). Cognitive remediation therapy in schizophrenia: randomised controlled trial. British Journal of Psychiatry 190, 421427.Google Scholar
Wykes, T, Reeder, C (2005). Cognitive Remediation Therapy for Schizophrenia: Theory and Practice. Routledge: London.Google Scholar
Wykes, T, Steel, C, Everitt, B, Tarrier, N (2008). Cognitive behavior therapy for schizophrenia: effect sizes, clinical models, and methodological rigor. Schizophrenia Bulletin 34, 523537.Google Scholar
Figure 0

Fig. 1. Study design. CBT, Cognitive–behavioural therapy; PSYRATS, Psychotic Symptoms Rating Scale; CBTp, cognitive–behavioural therapy for psychosis.

Figure 1

Table 1. Critical features of CIRCUITS cognitive remediation (Wykes & Reeder, 2005)

Figure 2

Fig. 2. Consolidated Standards of Reporting Trials (CONSORT) flow diagram for trial participants. SC, Social contact; CR, cognitive remediation; CBTp, cognitive–behavioural therapy for psychosis.

Figure 3

Table 2. Group demographic and clinical characteristics

Figure 4

Fig. 3. Mean auditory hallucination (AH) and Delusion subscale scores by visit for each allocation group. PSYRATS, Psychotic Symptoms Rating Scale.

Supplementary material: File

Drake et al. Supplementary Material

Supplementary Material

Download Drake et al. Supplementary Material(File)
File 159.7 KB