Introduction
Improving functional outcomes and decreasing disability rates in schizophrenia is a prevalent goal of current clinical research. Many studies are focusing on improving neurocognitive, daily, social, and community functioning within this population by using skills-training groups. Skills-training interventions allow clinicians to provide individualized instruction, feedback, and reinforcement to patients, which have been shown to have both short- and long-term benefits in everyday and social functioning (for e.g., Granholm, Holden, Link, McQuaid, & Jeste, Reference Granholm, Holden, Link, McQuaid and Jeste2012; Patterson et al., Reference Patterson, Mausbach, McKibbin, Bucardo, Jeste and Goldman2006). The ability for scientists to provide tools for clinicians to use to assist with determining whether patients will put forth sufficient effort to engage in the treatment protocol (i.e., attend group sessions, complete homework, practice skills learned) is crucial to subject retention and the evaluation of treatment success rates.
Group attendance, specifically, has been shown to improve treatment outcomes in psychiatric populations. Attending a group provides face-to-face interactions between the patient and clinician, thereby enabling the patient to receive instruction and reinforcement. In addition, attending a skills-training intervention allows the patient to engage in behavioral rehearsal, such as role play activities, as well as establish functional and social goals (Bellack, Reference Bellack2004; Penn & Mueser, Reference Penn and Mueser1996). Another major benefit of group attendance is that it provides individuals the opportunity to master learned skills via homework assignments. In other words, people who attend group have a greater chance of completing homework assignments, which allows them to practice the skills they have learned outside of group and receive feedback on their homework during subsequent groups. Meta-analytic studies have demonstrated that homework compliance, in particular, is related to improved treatment outcomes (Kazantzis, Deane, & Ronan, Reference Kazantzis, Deane and Ronan2000; Mausbach, Moore, Roesch, Cardenas, & Patterson, Reference Mausbach, Moore, Roesch, Cardenas and Patterson2010). It is important to note, however, that despite evidence underscoring the importance of attendance in treatment success, the ability to predict who will put forth adequate effort and actually attend treatment has not been extensively researched.
The most reliable and efficient way to assess effort is a constant debate in the literature. Research has consistently demonstrated that the use of clinician impressions alone are not sufficient for accurately detecting questionable effort and response bias during neurocognitive testing (Faust et al., Reference Faust, Guilmette, Hart, Arkes, Fishburne and Davey1988; Heaton, Smith, Lehman, & Vogt, Reference Heaton, Smith, Lehman and Vogt1978). Symptom validity tests (SVTs), such as the Test of Memory and Malingering (TOMM; Bianchini, Mathias, & Greve, Reference Bianchini, Mathias and Greve2001), are effective at identifying feigned cognitive impairments and are currently the industry standard for assessing effort. However, some have described these tests as lengthy, time consuming and impractical (Silverberg, Wertheimer, & Fichtenberg, Reference Silverberg, Wertheimer and Fichtenberg2007). As a way to simultaneously screen for cognitive impairment and inadequate effort, Silverberg et al. (Reference Silverberg, Wertheimer and Fichtenberg2007) developed the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) Effort Index (EI).
The RBANS EI is the first empirically derived measure of questionable effort that is embedded within a cognitive screening battery. The RBANS EI is comprised of the Digit Span and List Recognition subtests from the RBANS, subtests which have been found to be particularly sensitive to a participant's level of effort (Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007). Scores on each of the two subtests are inversely weighted (i.e., Digit Span raw score between 8 and 16 receives weighted score of 0; Digit Span raw score of 7 receives weighted score of 2) and then summed to create an overall score ranging from 0 to12, so that higher EI scores are thought to be reflective of poorer effort. In the original study, a heterogeneous clinical sample was used to identify an appropriate cut-off score for the RBANS EI to detect insufficient effort, then the EI was validated using a sample of genuine versus feigned traumatic brain injury patients (Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007). Still a relatively new measure of effort, only six studies have been published examining its reliability and validity (Armistead-Jehle & Hansen, Reference Armistead-Jehle and Hansen2011; Duff et al., Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011; Hook, Marquine, & Hoelzle, Reference Hook, Marquine and Hoelzle2009; O'Mahar et al., Reference O'Mahar, Duff, Scott, Linck, Adams and Mold2012; Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007; Young, Baughman, & Roper, Reference Young, Baughman and Roper2012). These studies conclude (although inconsistently) that scores of 3 or higher are indicative of low effort. However, no studies have yet to examine the RBANS EI in patients with schizophrenia. As a cognitive screening battery, the RBANS has demonstrated good reliability and validity in patients with schizophrenia (Gold, Queern, Iannone, & Buchanan, Reference Gold, Queern, Iannone and Buchanan1999; Hobart, Golberg, Bartko, & Gold, Reference Hobart, Golberg, Bartko and Gold1999), and similar psychometric properties for the RBANS EI in this population should theoretically and conceptually exist.
There were two aims to the present study. The first aim was to examine the relationship between the RBANS EI scores and group attendance in a skills-training treatment study. We predicted that patients whose RBANS EI were indicative of suboptimal effort would attend significantly fewer group sessions than patients whose RBANS EI was within normal limits. Given the known associated cognitive impairment among individuals with schizophrenia, we explored this relationship using the proposed cutoff score of 3 as well as a more conservative cutoff of 4, which may be more appropriate for use in this population and therefore a better predictor of treatment adherence. Indeed, there is a debate in the literature regarding the exact RBANS EI cutoff score that should be used as a valid indicator of suboptimal effort in cognitively impaired individuals, and to date no “gold standard” exists (Barker, Horner, & Bachman Reference Barker, Horner and Bachman2010; Duff et al., Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011; Hook et al., Reference Hook, Marquine and Hoelzle2009; Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007). Some studies consider RBANS EI scores greater than 3 to be questionable, while other studies suggest the EI score as low as 1 may be able to discriminate between participants who have actual cognitive impairment versus those whose score is low due to insufficient effort. For example, Duff et al. (Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011) and Hook et al. (Reference Hook, Marquine and Hoelzle2009) independently found that a cutoff score of 3 resulted in inflated RBANS EI “failure” effort rates in large heterogeneous samples of geriatric patients (i.e., medically ill older adults, nursing home residents, patients with amnestic mild cognitive impairment and probable Alzheimer's disease, and patients referred for neuropsychological evaluations). Specifically, both studies found that approximately one-third of their samples were inaccurately identified as providing poor effort using the proposed cutoff score. In an attempt to tease apart the effects of poor effort from disease pathology, we also examined whether psychiatric symptoms and overall cognitive functioning would be predictive of group attendance.
The second aim of the study was to examine the relationships between the RBANS EI and cognitive functioning, psychiatric symptoms, and mood in a schizophrenic sample. We hypothesized that patients whose RBANS EI was indicative of suboptimal effort would have worse cognitive functioning, greater positive symptoms, negative symptoms, and depressed mood.
Method
Participants
Participants were 128 non-litigating, ethnically diverse, community dwelling middle-aged and older adults with a DSM-IV (American Psychiatric Association, 2000) classification of schizophrenia or schizoaffective disorder. All 128 participants were enrolled in the Skills Training and Empowerment Program (STEP) study, a randomized clinical trial examining the effectiveness of two skills-training psychosocial interventions aimed at improving functional outcomes in middle-aged and older adults with schizophrenia and schizoaffective disorders. Participants were recruited from board and care (B&C) and day treatment facilities throughout San Diego County. To be eligible for the study, participants were required to be 40 years of age or older and have a physician-determined diagnosis of schizophrenia or schizoaffective disorder (verified by chart review, as diagnosed by the patient's psychiatrist). Exclusion criteria included a primary DSM-IV diagnosis other than schizophrenia or schizoaffective disorder, including dementia, and if participants were judged to be at risk for suicide, were unable to complete the assessment battery, or were currently participating in another psychosocial intervention or pharmacotherapy study. Approval for both studies was provided by the University of California, San Diego Institutional Review Board. All participants were administered the UCSD-Brief Assessment of Capacity to Consent (UBACC; Jeste et al., Reference Jeste, Palmer, Appelbaum, Golshan, Glorioso, Dunn and Kraemer2007) to determine whether participants had the capacity to provide informed consent before enrollment.
Once enrolled in the trial, participants were randomly assigned to one of two 24-week intervention conditions aimed at improving everyday living and social skills in middle-aged and older people with schizophrenia via role-play and practice of specific skills (e.g., planning and organization, money management, medication management, using public transportation). In the first condition, patients received the Functional Adaptation Skills Training (FAST) consisting of 24 two-hour weekly group sessions. In a previous RCT (Patterson et al., Reference Patterson, Mausbach, McKibbin, Bucardo, Jeste and Goldman2006), FAST was associated with improvement in performance-based measures of functioning (Patterson et al., Reference Patterson, Mausbach, McKibbin, Bucardo, Jeste and Goldman2006, Reference Patterson, McKibbin, Taylor, Goldman, Davila-Fraga, Bucardo and Jeste2003). Despite these positive findings, the median number of sessions attended was 13 of 24, and approximately 20% of participants attended fewer than 5 sessions. In addition, the use of skills outside of sessions was suboptimal. Post hoc analyses revealed approximately 68% of participants reported using the skills “not much” outside of the classroom setting and only 5% reporting using the skills “a lot.”
In the second condition, patients were enrolled in the Skills Training and Empowerment Program (STEP; Cardenas et al., Reference Cardenas, Abel, Bowie, Tiznado, Depp, Patterson and Mausbach2012; Depp et al., Reference Depp, Mausbach, Granholm, Cardenas, Ben-Zeev, Patterson and Jeste2010). STEP was developed to reduce the number of sessions of FAST as well as to enhance the use of skills in the natural environment by offering 12 two-hour biweekly groups and 12 cellular phone individual sessions (2 phone calls every 2 weeks between group sessions) with a therapist. STEP participants received their own basic cell phone on which they received 20-min phone calls from a therapist. Every call included a standard agenda: (a) “Check in”, in which counselors summarizes for participants the agenda for the cell-phone session; (b) inquiry on how the participant is regarding his/her well-being, emotions, and symptoms; (c) brief review of skills taught in the previous group session; (d) reminder of homework assignment, and assessment on whether they practiced the skills they were taught and the progress toward a behavioral goal they selected for the week; (e) assessment of barriers to practicing skills and achieving personal goals; and (f) reinforce achievements and problem-solve barriers and, if necessary, develop remedial measures so the participants can practice skills and make progress toward their goals.
Participants in both groups completed identical assessments before the initiation of treatment as well as 6-month, 12-month, and 18-month follow-up assessments and were compensated for assessment visits. No compensation was provided for treatment visits. Baseline data were used in this study. Given that problems with transportation are common among older adults with schizophrenia, the skills-training groups were delivered both at the research center (N = 3) and in the community at club houses (N = 26), board & care facilities (N = 71), and adult day programs (N = 28) by masters’ level therapists, who also tracked participant attendance.
Measures
The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS)
The RBANS (Randolph, Reference Randolph1998) is a brief cognitive screening battery consisting of 12 subtests which are used to create Index scores in the following five cognitive domains: Immediate Memory, Visuospatial/Constructional Skills, Language, Attention, and Delayed Memory. A total score is created by summing the five index scores which are thought to represent one's current neuropsychological status. The RBANS was created as a screening tool for use in neurologically impaired individuals and was normed with Caucasians and African-Americans between the ages of 20 and 89 years. A total score of 100 is considered average with a standard deviation (SD) of 15. Thus, participants scoring one SD below average are considered mildly impaired, two SDs below is moderately impaired, and three SDs below is indicative of severe impairment. Good reliability and validity exists for the RBANS in both geriatric (Gontkovsky, Beatty, & Mold, Reference Gontkovsky, Beatty and Mold2004) and schizophrenia populations (Gold et al., Reference Gold, Queern, Iannone and Buchanan1999; Hobart et al., Reference Hobart, Golberg, Bartko and Gold1999).
The EI was calculated using the procedures described by Silverberg et al. (Reference Silverberg, Wertheimer and Fichtenberg2007). Scores on the Digit Span and List Recognition subtests are inversely weighted and summed so that lower scores are converted to higher scores on the EI. Each subtest receives a weighted score between 0 to 6, then the two weighted scores are summed to produce an overall EI score ranging from 0 to 12. Higher scores are suggestive of poorer effort.
Positive and Negative Syndromes Scale (PANSS)
The PANSS (Kay, Fiszbein, & Opler, 1987) is a 30-item structured interview rating scale used to assess positive (e.g., hallucinations, delusions), negative (e.g., impaired attention, blunt affect, social withdrawal), and general psychopathology (e.g., uncooperativeness, disorientation, lack of judgment and insight) symptoms of schizophrenia. Total scores on the positive and negative symptom scales range from 7 to 49, with higher scores indicating greater symptomatology. General psychopathology scores range from 16 to 112 and higher scores indicate greater cognitive impairment/more psychopathology. Good internal consistency and test–retest reliability have been demonstrated with this measure (Kay, Fiszbein, & Lewis Reference Kay, Fiszbein and Lewis1987; Mass, Schoemig, Hitschfeld, Wall, & Haasen, Reference Mass, Schoemig, Hitschfeld, Wall and Haasen2000).
Calgary Depression Scale for Schizophrenia (CDSS)
The Calgary Depression Scale for Schizophrenia (CDSS; Addington & Addington, Reference Addington and Addington1990) was designed specifically for use in schizophrenia to assess depressive symptoms independent of symptoms of psycopathology. The CDSS is a nine-item questionnaire assessing the following symptoms of depression: depressed mood, hopelessness, self-depreciation, guilty ideas of reference, pathological guilt, morning depression, early wakening, suicide, and observed depression. Scores awarded for responses to each item range from 0 to 3, with a global score range of 0–27 points. There are eight structured questions and one clinician-rated observation.
Data Analysis
Our first approach to the data was to generate an RBANS EI frequency distribution in our sample. The effects of age and education were also examined in regard to performance on the EI. We conducted receiver operating characteristic (ROC) curves to (1) determine the ability of the EI to discriminate between participants with varying levels of group attendance, and (2) to determine what cutoff score, if any, provided the best sensitivity and specificity in distinguishing between groups. Youden's Index value, which is one minus specificity subtracted from the sensitivity (Fluss, Faraggi, & Reiser, Reference Fluss, Faraggi and Reiser2005; Loong, Reference Loong2003), was used to determine the number of groups attended with the best sensitivity and specificity. Next, a logistic regression analysis was conducted to examine the overall impact of the EI, RBANS total score, and PANSS symptoms on the prediction of high versus low treatment attendance. For this analysis, high attendance was classified as 80% attendance which was the value with the best sensitivity and specificity. Lastly, to assess convergent validity of the EI with cognition, psychiatric symptoms, and mood, Pearson bivariate correlations were performed. To control for familywise error, the critical value was Bonferroni-adjusted to alpha = 0.003 (0.05/15 total comparisons) for correlational analyses.
Results
Mean overall age was 51.9 (SD = 7.2) and mean years of education was 12.3 (SD = 2.34). A total of 69 participants (53.1%) had attendance greater than or equal to 75%. The only significant differences found based on condition (FAST vs. STEP) was for age and race; no other demographic or clinical differences existed between the two conditions (Table 1). As a whole, participants had mean RBANS total scaled scores of 63.84 (SD = 12.33; range = 42–106), which falls in the cognitively impaired range. We calculated RBANS EI scores according to procedures outlined in Silverberg et al. (Reference Silverberg, Wertheimer and Fichtenberg2007). RBANS EI scores ranged from 0 to 11 (M = 2.16; SD = 2.55), with 22.66% (n = 29) of our sample obtaining RBANS EI scores greater than the recommended cutoff of 3, suggesting low effort. By comparison, 31% of medically ill older adults (RBANS Total Score: M = 68.30; SD = 13.82; n = 44) (Hook et al., Reference Hook, Marquine and Hoelzle2009), 37% of nursing home residents (RBANS Total Score: M = 73.1; SD = 11.90; n = 229), and 33% of patients with probable Alzheimer's disease (RBANS Total Score: M = 65.9; SD = 5.90; n = 126) (Duff et al., Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011) scored above the recommended cutoff of 3. However, only 3% of cognitively intact geriatric individuals (RBANS Total Score: M = 95.7; SD = 13.3; n = 796) (Duff et al., Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011) scored above the cutoff. Thus, although our sample appeared to have similar overall cognitive impairment relative to these populations, they appeared to perform better in terms of effort. Neither age (r = 0.07; p = .45) nor education (r = 0.04; p = .64) were related to RBANS EI scores in our sample.
Table 1 Comparison of FAST and STEP group participants
Note. RBANS EI = Repeatable Battery for the Assessment of Neuropsychological Status Effort Index; PANSS Pos = Positive and Negative Syndrome Scale - Positive Symptoms; PANSS Neg = Positive and Negative Syndrome Scale - Negative Symptoms; PANSS GP = Positive and Negative Syndrome Scale - General Psychopathology.
at-test for continuous variables; Chi-square for categorical variables.
Prediction of Group Attendance
We used area under the ROC curves (AUC) with 95% confidence intervals to determine the ability of the RBANS EI to differentiate participants based on various levels of group attendance. With an estimated AUC of 0.57 (95% confidence interval [CI], 0.47–0.67; p = .21), an EI cutoff of <3 was not able to significantly predict group attendance. However, using a more conservation cutoff of <4, the EI did provide significant predictive ability between group attendance, with an estimated AUC of 0.63 (95% CI, 0.52–0.74, p = .04; see Figure 1). Table 2 provides information on the sensitivity, specificity, and Youden's Index (Fluss et al., Reference Fluss, Faraggi and Reiser2005) for varying amounts of groups attended, including 50%, 60%, 70%, 80%, and 90%. We considered dividing the participants into dichotomous groups of high versus low attendance, but the literature on the exact number of sessions deemed necessary for clinical improvement varies greatly by study. Therefore, we chose to provide data for various cutoffs in an attempt to help clinicians determine how effective the EI will be in predicting treatment attendance based on the nonlinear nature of the relationship between therapy and clinical benefit (Reardon, Cubkrowicz, Reeves, & Joiner, Reference Reardon, Cubkrowicz, Reeves and Joiner2002). Fifty percent was chosen as the lowest value to present, as there is a consensus in the literature that less than 50% attendance in treatment is deemed disengaged (Baekeland & Lundwall, Reference Baekeland and Lundwall1975). In our analyses, sensitivity refers to the percentage of participants with greater than a particular amount of attendance (i.e., attended more than 70% of total classes) who were correctly identified by an EI score of less than four. Specificity refers to the percentage of participants with less than a particular attendance rate who were correctly identified by an EI score greater than or equal to four. As seen in Table 2 and Figure 1, the EI was most accurately able to differentiate between participants who attended more or less than 80% of classes with a Youden's index value of 0.36.
Fig. 1 Receiver Operating Characteristics (ROC) curve for prediction of group attendance using the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) Effort Index (EI) cutoff of 4.
Table 2 Sensitivity, specificity, and Youden's Index of various percentages of groups attended using RBANS EI cutoff of 4
RBANS EI = Repeatable Battery for the Assessment of Neuropsychological Status Effort Index.
aMaximum Sensitivity and Specificity.
Logistic regression was conducted to explore whether psychiatric symptoms and overall cognitive functioning were able to discriminate between high and low treatment engagement. For these analyses, we used 80% to differentiate between high and low attendance and an EI cutoff of 4. The EI was a significant predictor of attendance (r = 0.59; p < .01), whereas positive symptoms (r = −0.02; p = .25), negative symptoms (r = 0.07; p = .51), general psychopathology (r = 0.01; p = .26), and overall cognitive functioning (r = 0.00; p = .73) were not. These findings suggest that the RBANS EI assesses a construct that is not being measured by psychiatric symptoms or overall cognitive functioning.
Relationships With RBANS EI, Cognitive Functioning, and Symptoms
As can be seen in Table 3, the RBANS EI was negatively correlated with cognitive functioning (RBANS total scores) and positively correlated with psychopathology (PANSS negative and PANSS general psychopathology).
Table 3 Bivariate correlations with RBANS EI
RBANS EI = Repeatable Battery for the Assessment of Neuropsychological Status Effort Index; UPSA = UCSD Performance-Based Skills Assessment; SSPA = Social Skills Performance Assessment; PANSS Pos = Positive and Negative Syndrome Scale - Positive Symptoms; PANSS Neg = Positive and Negative Syndrome Scale - Negative Symptoms; PANSS GP = Positive and Negative Syndrome Scale – General Psychopathology; CDSS = Calgary Depression Scale for Schizophrenia.
an = 56.
*p < .003.
Discussion
This study was the first to examine the RBANS Effort Index (EI) (Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007) in a sample of patients with schizophrenia. Results suggest that there may be a significant relationship between high scores on the RBANS EI (indicative of suboptimal effort), worse cognitive functioning, more negative symptoms, and greater general psychopathology. Furthermore, as hypothesized, analyses indicated that the RBANS EI can be a useful predictor of group attendance in a research setting.
Both Hook et al. (Reference Hook, Marquine and Hoelzle2009) and Duff et al. (Reference Duff, Spering, O'Bryant, Beglinger, Moser, Bayless and Scott2011) previously found that among medically ill and cognitive impaired older patients a cutoff of 3 resulted in inflated RBANS EI “failure” effort rates, with one-third of their participants inaccurately identified as providing poor effort using the proposed cutoff score. We found that a more conservative EI cutoff of 4 had greater attendance predictive abilities than a lower cutoff. Given the age and overall levels of cognitive impairment in our sample, using a conservative cutoff which adjusts for these factors seemed clinically appropriate (Young et al., Reference Young, Baughman and Roper2012).
Also as predicted, global cognitive status, as measured by the RBANS total score, and psychiatric symptoms were unable to predict group attendance in our sample, indicating that the EI is measuring a different construct than the cognitive effects of the disease pathology. Other studies have also teased apart the effects of effort from cognition in individuals with schizophrenia. For example, psychotic patients have been found to score in the “passing” range, as well as significantly better than forensic samples, on a well-established measure of suspect effort: the Test of Memory Malingering (TOMM; Duncan, Reference Duncan2005; Gierok, Dickson, & Cole, Reference Gierok, Dickson and Cole2005; Tombaugh, Reference Tombaugh1996). Additionally, Kertzman et al. (Reference Kertzman, Grinspan, Birger, Shliapnikov, Alish, Ben Nahum and Kotler2006) demonstrated that, in a forensic setting, schizophrenia patients with active psychosis performed significantly better on two independent computerized tests of effort than schizophrenia patients who feigned acute psychotic symptoms and cognitive impairment. In regard to the EI specifically, Silverberg et al. (Reference Silverberg, Wertheimer and Fichtenberg2007) administered both the EI and the TOMM to healthy controls, clinical malingers, and simulated malingers, and found that, when they partialed out effects of TOMM performance from the relationship between RBANS EI and “other cognitive functions” on the RBANS, the previously significant relationship between EI and other cognitive functions became non significant. That the EI was able to predict group attendance when cognitive functioning and symptoms were not may have important implications in the development of more effective recruitment strategies for treatment interventions. Considering the high attrition rate typically reported in intervention studies targeting this population (Harding et al., Reference Harding, Torres-Harding, Bond, Salyers, Rollins and Hardin2008), discovering a potential screening tool that provides further insight into which participants will put forth adequate effort during training may prove to be helpful in the successful retention of participants. Moreover, an effort screening tool such as the RBANS EI may provide a more accurate interpretation of patient outcomes and progress, which are typically based on normative data and assume that the patient is answering questions to the best of his/her ability.
In the sample used in this study, motivation to purposefully feign cognitive impairment was likely low. However, low motivation and apathy are characteristic features of schizophrenia. Therefore, it can reasonably be inferred that an unwillingness to make an attempt to perform well on the task, and not malingering per se, are likely what are being assessed with the RBANS EI and contributing to worse performance on measures of everyday and social functioning. This finding is consistent with prior research by Foussias, Mann, Zakzanis, van Reekum, and Remington (Reference Foussias, Mann, Zakzanis, van Reekum and Remington2009), showing that motivational deficits accounted for approximately 74% of the variance in functioning in patients with schizophrenia. Konstantakopoulos et al. (Reference Konstantakopoulos, Ploumpidis, Oulis, Patrikelis, Soumani, Papadimitriou and Politis2011) found similar results and described apathy as the most robust predictor of current psychosocial functioning, accounting for approximately 70% of the variance in functioning.
It is important to note that although numerous studies use the RBANS to assess neurocognitive functioning in a variety of domains, the EI index, although embedded in the test, is seldomly evaluated. Considering some of the literature supporting the good discriminability and psychometric properties of the this index (Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007), it is plausible that it can be used to enhance the interpretation of scores on some of the RBANS subtests. For example, if a participant scores exceptionally low on the RBANS subtests but has a high score on the EI and appears to perform above average on other neuropsychological measures, the low score on the RBANS subtests may be indicative of poor effort rather than actual cognitive deficits. However, more research on the RBANS EI is warranted before the index can be considered a routinely used reliable marker of cognitive effort. Moreover, it is important to note that the results of a single index score are not necessarily meant to stand alone or be applied dogmatically. Instead, they should more appropriately be considered one of many potentially useful aides available to clinicians to help determine a patient's effort level.
Although there are notable strengths to the study, there are several limitations that must be acknowledged. A diagnosis of schizophrenia or schizoaffective disorder was determined by chart diagnosis, as diagnosed by the patient's psychiatrist, and a structured diagnostic interview would be a more reliable way to verify diagnostic accuracy. Another limitation is the fact that the extant literature on the RBANS EI is relatively scarce and, therefore, a consensus on a standard cutoff score that generalizes across samples has yet to be determined. Therefore, this study compared a cutoff score of 3, which has demonstrated some convergent evidence of questionable effort (Silverberg et al., Reference Silverberg, Wertheimer and Fichtenberg2007), and a more conservative cutoff of 4, a cutoff that has not previously been validated. This study did not have an external criterion measure of performance effort to compare with the RBANS EI, which would have helped to validate a cutoff of 4 or help determine if an even more conservative cutoff would be more appropriate in this population. Measures of effort should not be used in isolation due to risk of false negatives, and further investigation that uses multiple cognitively based effort measures [e.g., Word Memory Test (Gorissen, Sanz, & Schmand, Reference Gorissen, Sanz and Schmand2005) and Reliable Digit Span (Greiffenstein, Baker, & Gola, Reference Greiffenstein, Baker and Gola1994)] and compares the scores of the effort indices to one another as well as to some of the outcome variables we reported in our results may extend this study's findings.
In addition to being a potential predictor of overall functioning, assessing a patient's effort or level of motivation may have clinical utility by operating as a guide in the development of more effective rehabilitation programs. For example, results suggest that a patient with high motivation thrives in a rigorous, intensive skill learning class focused on improving everyday functional skills. It would be interesting to assess whether a patient with low motivation excels in a less rigorous program or a program with different aims, such as social skills training or cognitive rehabilitation. Our results suggest that using measures that include some form of an effort index can help clinicians and scientists determine, on an individual level, what type of treatment will be a good fit for their patients.
In sum, there is a paucity of literature specifically looking at effort testing and the role effort may play in determining a variety of patient functional outcomes. This study was designed in an attempt to help address this gap in the literature. The importance of developing cogent models that help explain the role of effort in testing stems from research showing that insufficient effort may threaten the validity of neuropsychological test performance and may further obscure group differences on cognitive tasks (Green, Lees-Haley, & Allen, Reference Green, Lees-Haley and Allen2002). Furthermore, as reported in this study, the magnitude of a patient's effort has the ability to influence treatment outcomes such as everyday functioning, social functioning, and rate of participation and should therefore be adequately measured and taken into consideration when making a judgment on the reliability of data collected and potentially in the selection of participants for intervention programs.
Acknowledgments
This work was supported by the National Institute of Mental Health [R01 MH084967, T32MH019934, and 5P30MH066248]. The authors affirm that there exists no conflict of interest.
