Datasets

The characteristics of the different datasets are listed in Table 1 and Table 2. Datasets 1–3 were obtained at the Netherlands Cancer Institute (NKI). Datasets 4–6 were obtained at M.D. Anderson Cancer Center (MDACC). None of the datasets contain overlapping data.

Table 1 Characteristics of patients in NKI datasets

¹Education is based on Verhage education scores 1–7 (Verhage, Reference Verhage1964), corresponding with the following US years of education; 1: 1–5 years; 2: 6 years; 3: 7–8 years; 4: 7–9 years; 5: 7–10 years; 6: 7–16 years; 7: 17–20 years.

²Fatigue based on EORTC QLQ-C30, EORTC = European Organization for Research and Treatment of Cancer: EORTC QLQ-C30 is a health-related quality of life questionnaire (Aaronson et al., Reference Aaronson, Ahmedzi, Bergman, Bullinger and Cull1993). Score on the Fatigue scale ranges from 0 to 100, with a higher score meaning more bothered by complaint.

³Anxiety and Depression based HSCL-25, HSCL-25 = the Hopkins symptom checklist-25 (Hesbacher et al., Reference Hesbacher, Rickels, Downing and Stepansky1978). Scores range from 0 to 100; a higher score indicates more complaints.

CTC = cyclophosphamide, thiotepa, carboplatin; FEC = 5-fluorouracil, epirubicin, cyclophosphamide; CMF = cyclophosphamide, methotrexate, 5-fluorouracil; CT = chemotherapy; TAM = tamoxifen; EXE = exemestane.

Table 2 Characteristics of patients in MDACC datasets

¹ Fatigue level on a 0-10 scale with 10 being the worst.

²Anxiety based on State-Trait Anxiety Inventory (z-scale) (Spielberger et al., Reference Spielberger, Gorsuch, Lushene, Vagg and Jacobs1983), Minnesota Multiphasic Personality Inventory (MMPI) (Greene, Reference Greene1991) scale 7 (T-scores) and, Beck Anxiety Inventory (Beck, Reference Beck1990). A higher score indicated a higher level of anxiety.

³Depression based on the Beck Depression Inventory (Beck, Reference Beck1996) and, MMPI (Greene, Reference Greene1991) scale 2 (T-scores).

FAC = 5-fluorouracil, doxorubicin, cyclophosphamide; TAM = tamoxifen.

1. 1. Dataset 1. The first dataset originated from a prospective study performed by Schagen et al. (Reference Schagen, Muller, Boogerd, Mellenbergh and van Dam2006). In this study, changes in cognitive performance were examined in three different breast cancer groups and a healthy control group. The breast cancer patient groups consisted of (1) patients who adjuvantly received four cycles of standard-dose chemotherapy (FEC) followed by one cycle of high-dose chemotherapy (CTC) (labeled “CTC”, n = 28); (2) patients who received five cycles of standard-dose FEC chemotherapy (labeled “FEC”, n = 39); and (3) a breast cancer control group (labeled “Control”, n = 57) comprised by stage-1 breast cancer patients for whom systemic therapy was not part of the treatment strategy. The breast cancer patients receiving chemotherapy were randomized between either high-dose or standard dose-treatment (Rodenhuis et al., Reference Rodenhuis, Bontenbal, Beex, Wagstaff, Richel, Nooij and de Vries2003). Patients who received chemotherapy also received endocrine therapy (20 mg a day for a period of 2 years). All patients underwent neuropsychological testing before chemotherapy and 6 months after treatment over a 12-month interval. The healthy control group underwent repeated testing over a 6-month interval.

2. 2. Dataset 2. This is the dataset of a prospective study in which the influence of endocrine therapy on cognitive functioning was examined in postmenopausal patients with breast cancer (Schilder et al., Reference Schilder, Seynaeve, Beex, Boogerd, Linn, Gundy and Schagen2010). Patients were randomized to either tamoxifen (n = 80) or exemestane (n = 99) and underwent neuropsychological testing after breast surgery and before the start of endocrine treatment. Follow-up assessment was conducted after 1 year of endocrine treatment. A healthy control group (n = 120) underwent the same assessments with a similar time interval.

3. 3. Dataset 3. This dataset describes four patient groups that were examined on average 2 years after completion of treatment (Van Dam et al., Reference Van Dam, Schagen, Muller, Boogerd, vd Wall, Droogleever Fortuyn and Rodenhuis1998; Schagen et al., Reference Schagen, van Dam, Muller, Boogerd and Lindeboom1999). In this study, the cognitive status of breast cancer patients treated with various cytotoxic regimens as part of an adjuvant therapy strategy was examined. A group of patients treated with high-dose adjuvant chemotherapy (CTC group; n = 34), two groups treated with conventional dose chemotherapy (FEC group; n = 36, CMF group; n = 39). CTC and FEC patients and a select number of randomized CMF patients (n = 20) also received endocrine treatment with Tamoxifen for 2 years. A group of breast cancer patients without any systemic treatment was also included (n = 34).

4. 4. Dataset 4. The fourth dataset stems from a prospective longitudinal study performed by Wefel et al. (Reference Wefel, Saleeba, Buzdar and Meyers2010). In this study, the incidence, nature, and chronicity of cognitive dysfunction in 42 breast cancer patients undergoing adjuvant chemotherapy according to the FAC regimen with or without paclitaxel was examined. Patients were tested before, during, and at two time points after chemotherapy.

5. 5. Dataset 5. This set contains data from a prospective longitudinal study performed by Wefel et al. (Reference Wefel, Lenzi, Theriault, Davis and Meyers2004). In this study, cognitive changes of breast cancer patients who received chemotherapy with FAC with or without methotrexate and vinblastine was examined over time (n = 18). Patients underwent neuropsychological testing before and at three time points after completion of chemotherapy.

6. 6. Dataset 6. The last dataset originates from a prospective longitudinal study performed by Wefel (2002) on cognitive changes in breast cancer patients treated with endocrine therapy (n = 62). Patients underwent neuropsychological testing after their primary medical treatment (e.g., surgery, radiation, chemotherapy) and before starting Tamoxifen. Follow-up assessment was performed twice after starting endocrine treatment.

We searched Pubmed, EMBASE, and PsychInfo (January 1980–January 2013) and reference lists of articles in the English language for embedded PVTs that corresponded with the neuropsychological tests used in the six studies. Search terms were: (noncredible performance) AND (embedded symptom or performance validity) AND (malingering) AND (embedded effort) in combination with the neuropsychological instruments used in our datasets.

The following tests were determined to contain embedded PVT indices: California Verbal Learning Test (CVLT), Controlled Oral Word Association (COWA), WAIS-III subtest Digit Span, Rey-Osterrieth Complex Figure Test (ROCFT), Rey Auditory Verbal Learning Test (RAVLT), Stroop test, and Trail Making Test (TMT). We also examined the Fake Bad Scale (FBS) from the Minnesota Multiphasic Personality Inventory (MMPI).

Table 3 summarizes the identified performance validity indices, the operational definitions of noncredible performance for each index, and their sensitivity and specificity. For each test, the embedded PVT index was evaluated based on cutoffs previously described in published literature. Additionally, measures of fatigue, depression, and anxiety were obtained in most of the study datasets and were examined in relation to PVT performance.

Table 3 Characteristics of the studies from which cutoffs were determined

¹Study participants include participants instructed to simulate, patients with and without credible performance, and patients evaluated in context of litigation. LTCR = Long term cued recall; NA = not available

Measures

1. 1. California Verbal Learning Test. The Dutch version of the CVLT (Delis, Kramer, Kaplan, & Ober, Reference Delis, Kramer, Kaplan and Ober1987) was used in dataset 1. The indices calculated in the Dutch version are different compared to the indices in the original CVLT (Mulder, Dekker, & Dekker, 1996). However, the administration and the data obtained are identical. Millis and Putman (Reference Millis and Putman1995) identified the following five indices suitable for examining noncredible performance: CVLT Total (sum of the scores on List A trial 1 to 5); Long Delay Cued Recall (number of correct responses on List A LDCR), Discriminability (score on the following function [1-(false positives+misses)/44] × 100); Recognition Hits (number of correct answers on List A Long Term Recognition, using the number of false positives and misses on List A Long Term Recognition) and Discriminant function score ((CVLT Total raw score)* (−0.00406) + (LDCR raw score)*(0.21099) + (Discriminability)*(0.04988) − 5.06399).

2. 2. Fluency. Various versions of a letter fluency test were used among the different datasets. In dataset 2, the Dutch letter combination DAT was used (Schmand, Groenink, & van den Dungen, Reference Schmand, Groenink and van den Dungen2008) that corresponds with the US FAS-test. In dataset 4 to 6, the letter combinations of the Multilingual Aphasia Examination were used (Benton, Hamsher, & Sivan, Reference Benton, Hamsher and Sivan1994). Curtis, Thompson, Greve, and Bianchini (Reference Curtis, Thompson, Greve and Bianchini2008) defined a cutoff of a T-score below 32 as an indicator of possible noncredible performance.

3. 3. Digit Span. Using the WAIS-III (Wechsler, Reference Wechsler1997) or WAIS-R (Wechsler, Reference Wechsler1981) Digit Span subtest in dataset 3 to 6, the Reliable Digit Span (RDS) was calculated. The RDS is the sum of the longest span forward and the longest span backward. An RDS score below 7 has been suggested as a cutoff for possible noncredible performance (Mathias, Greve, Bianchini, Houston, & Crouch, Reference Mathias, Greve, Bianchini, Houston and Crouch2002).

4. 4. Rey-Osterrieth Complex Figure Test. The ROCFT (Rey, Reference Rey1941) was partly administrated in datasets 3 and 6. Dataset 3 contained information about the copy score and the 3-min delay score of the ROCFT. Dataset 6 contained information only about the copy score. Reedy et al. (2002) identified a copy score of 26 or lower and/or a 3-min delay score of 10 or lower as an indication of possible noncredible performance.

5. 5. Rey Auditory Verbal Learning Test. Datasets 2 and 3 provided information about the recognition score of the RAVLT (van den Burg, Saan, & Deelman, Reference Van den Burg, Saan and Deelman1985). A recognition score of 9 or lower has been recommended as an indicator of possible noncredible performance (Boone, Lu, & Wen, Reference Boone, Lu and Wen2005).

6. 6. Stroop test. The Dutch datasets 1 to 3 contained information on the Stroop color-word test (Hammes, Reference Hammes1978). Based on a study by Arentsen et al. (Reference Arentsen, Boone, Lo, Goldberg, Cottingham, Victor and Zeller2013) in which the superiority was demonstrated of Word Naming trials and Color Naming trials over an inverted Stroop effect as an effective PV measure, we choose the cutoff of ≥66 s for Word Naming and ≥93 s for Color Naming as indicators of possible noncredible performance, maintaining adequate specificity and sensitivity. In the Arentsen paper, the Comalli version of the Stroop Color Word Test was used. This version is highly comparable to the version used in the Dutch datasets (i.e., identical number of trials and items). The Dutch version included 4 response options compared to 3 in the Comalli version.

7. 7. Trail Making Test (TMT). All datasets contained information on Part A and Part B of the TMT (Reitan, Reference Reitan1958). Iverson, Lange, Green, and Franzen (Reference Iverson, Lange, Green and Franzen2002) defined a completion time of 63 s or higher on Part A and/or a completion time of 200 s or higher on Part B as an indicator of possible noncredible performance.

8. 8. Hiscock and Hiscock. Hiscock and Hiscock (Reference Hiscock and Hiscock1989) developed a stand-alone PVT using the forced-choice method. One dataset contained information from a modified Hiscock & Hiscock test where 20 cards were shown for 5 s with a 5-s delay between presentation and recognition conditions. A score below chance is regarded as an indicator of possible noncredible performance.

9. 9. Minnesota Multiphasic Personality Inventory (MMPI). The FBS developed by Lees-Haley, English, and Glenn (Reference Lees-Haley, English and Glenn1991)and derived from items on the MMPI (Greene, Reference Greene1991) was intended to detect self-reported symptom magnification. Larrabee's (Reference Larrabee2003)regression formula was used to estimate the FBS score from the 370-item protocol for all patients in Dataset 5. A FBS score of ≥26 was used to identify possible symptom exaggeration (Nelson, Parsons, Grote, Smith, & Sisung, Reference Nelson, Parsons, Grote, Smith and Sisung2006).

10. 10. Years of education. In the Dutch datasets 1 to 3, Verhage education scores (Verhage, Reference Verhage1964) were used, according to Dutch standards (Bouma, Mulder, Lindeboom, & Schmand, Reference Bouma, Mulder, Lindeboom and Schmand2012). The US datasets 4 to 6 describe years of education.

11. 11. Fatigue. In datasets 1 to 3, the fatigue scale of the EORTC-QLQ-30 (Aaronson, Ahmedzi, Bergman, Bullinger, & Cull, Reference Aaronson, Ahmedzi, Bergman, Bullinger and Cull1993) was administrated. The fatigue scale consists of three items. Scores range from 0 to 100, with higher scores indicating more symptoms. In dataset 6 patients were asked to rate their fatigue level on a 0–10 scale with 10 being the worst.

12. 12. Anxiety and Depression. In datasets 1 to 3, the Hopkins Symptom Checklist-25 (HSCL-25) (Hesbacher, Rickels, Downing, & Stepansky, Reference Hesbacher, Rickels, Downing and Stepansky1978) was used to determine anxiety and depression. The HSCL-25 is a self-report measure of psychological distress. It includes an anxiety and depression subscale. The instructions inquire about the intensity of symptoms in the previous week. Answers are scored on a scale from 1 (never) to 4 (always). A higher score indicates greater distress.

In dataset 4, the State-Trait Anxiety Inventory (STAI) (Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, Reference Spielberger, Gorsuch, Lushene, Vagg and Jacobs1983) was used to determine anxiety. The STAI is a 20-item self-report measure to assess levels of situation-related anxiety. Items are rated on a 4-point Likert-type scale. The score is then converted into a Z-scale distribution based on comparisons with a healthy non-anxious sample. To assess depression, the Beck Depression Inventory (BDI) (Beck, Reference Beck1996) was used. The BDI is a 21-item self-report questionnaire measuring symptoms of depression. Items are rated on a 4-point scale. A higher score indicates greater depressive symptoms.

In dataset 5, the MMPI was used to determine anxiety and depression. MMPI scale-2 measures depression and MMPI scale-7 measures anxiety. Each scale is converted into a standardized T-score. A higher score indicates more symptoms.

In dataset 6, the BDI (Beck, Reference Beck1996) was used to determine depression. To determine anxiety the Beck Anxiety Inventory (BAI) (Beck, Reference Beck1990) was used. The BAI is a 21-item self-report measure to assess the cognitive aspects of anxiety. Items are rated on 4-point scale. A higher score indicates higher anxiety levels.

Statistical Analysis

The Statistical Package for Social Sciences (SPSS) version 20.0 was used for all statistical analyses. For each PVT index, a dichotomous variable was created indicating performance above or below the cutoff. The percentage of patients and controls with performances below the cutoff for each test and between group differences in these percentages were analyzed using χ² tests. Possible noncredible performance at the subject level was operationally defined as performance below the cutoff on three or more PVT indices. As discussed in Larrabee (Reference Larrabee2007), combining PVTs improves specificity with evidence that failure on three indicators essentially ensures no false positive classifications. We applied this strict criterion to the measures as we primarily had these available in the datasets, the cutoffs for the measures were not derived in our study population, the subjects in our studies did not have incentive for external gain, and we wished to minimize risk of falsely declaring noncredible performance (as opposed to cognitive dysfunction) in a population where there is evidence of brain dysfunction associated with treatment. Only one embedded PVT index was counted for tests that yielded multiple embedded PVT measures (Larrabee, Reference Larrabee2008) with preference given to including PVT indices below the cutoff. Logistic regression analyses were performed to examine group differences. Additional factors including age, education, fatigue, anxiety and depression were also examined. Assuming a low but non-negligible base rate of noncredible performance of 10%, likelihood ratios, pre- and post-test odds and posttest probabilities were calculated using the sensitivity and specificity of the applied cutoffs as depicted in the literature, according to the method described by Larrabee (Reference Larrabee2008). Using chained likelihood ratios, comparisons were made between posttest probabilities of the different combinations of PVTs in each dataset.