Hostname: page-component-7b9c58cd5d-dkgms Total loading time: 0 Render date: 2025-03-15T19:38:55.426Z Has data issue: false hasContentIssue false

Validity of the Dynamic Wisconsin Card Sorting Test for Assessing Learning Potential in Brain Injury Rehabilitation

Published online by Cambridge University Press:  10 November 2014

Hileen Boosman
Affiliation:
Brain Center Rudolf Magnus and Center of Excellence for Rehabilitation Medicine, University Medical Center Utrecht and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
Johanna M.A. Visser-Meily
Affiliation:
Brain Center Rudolf Magnus and Center of Excellence for Rehabilitation Medicine, University Medical Center Utrecht and De Hoogstraat Rehabilitation, Utrecht, The Netherlands
Tamara Ownsworth
Affiliation:
School of Applied Psychology and Griffith Health Institute, Griffith University, Australia
Ieke Winkens
Affiliation:
Maastricht University, Department of Psychiatry and Neuropsychology, School for Mental Health and Neuroscience, Maastricht, the Netherlands
Caroline M. Van Heugten*
Affiliation:
Maastricht University, Department of Psychiatry and Neuropsychology, School for Mental Health and Neuroscience, Maastricht, the Netherlands Maastricht University, Department of Neuropsychology and Psychopharmacology, Faculty of Psychology and Neuroscience, Maastricht, the Netherlands
*
Correspondence and reprint requests to: C. M. van Heugten, Maastricht University, School for Mental Health and Neuroscience, Department of Psychiatry and Neuropsychology, P.O. Box 616, 6200 MD Maastricht, The Netherlands. E-mail: c.vanheugten@maastrichtuniversity.nl
Rights & Permissions [Opens in a new window]

Abstract

The dynamic Wisconsin Card Sorting Test (dWCST) examines the effects of brief training on test performance where pre- to post-test change reflects learning potential. The objective was to examine the validity of the dWCST as a measure of learning potential in patients with acquired brain injury (ABI). A total of 104 patients with ABI completed the dWCST at rehabilitation admission. Performance of a subgroup (n=63) was compared to patients (n=28) who completed a repeated administration of the conventional WCST (rWCST). Furthermore, dWCST performance was compared between patients with ABI (n=63) and healthy controls (n=30) matched on gender, age, and education. Three learning potential indices were used: post-test score, gain score, and a group classification (decliners, poor learners, strong learners, high achievers). The median dWCST administration time was 30 min. The dWCST showed no floor or ceiling effects and the post-test and gain score were significantly intercorrelated. The pre-test score showed no significant associations with other neuropsychological tests. The learning potential indices were significantly associated with language and/or memory. In contrast to the dWCST group, the rWCST group showed no significant pre- to post-test improvement. There were significantly more poor learners in the rWCST group. Compared to controls, patients obtained similar gains, but significantly lower pre- and post-test scores for the dWCST. The ratio of poor learners between-groups was not significantly different. The results support the validity of the dWCST for assessing learning potential in patients with ABI. Further research is needed to investigate the predictive validity of the dWCST. (JINS, 2014, 20, 1–11)

Type
Research Articles
Copyright
Copyright © The International Neuropsychological Society 2014 

Introduction

During brain injury rehabilitation, patients re-learn old skills and learn new skills with the ultimate goal of optimizing social participation and well-being (Wade, Reference Wade2005). Although learning is essential for achieving rehabilitation gains, there is a lack of research-based guidelines on how best to assess learning potential in rehabilitation. A promising assessment method to evaluate learning potential that has been used in patients with acquired brain injury (ABI) and several other populations (e.g., schizophrenia, dementia), is dynamic testing (Calero & Navarro, Reference Calero and Navarro2004; Fernández-Ballesteros et al., Reference Fernández-Ballesteros, Botella, Zamarrón, Molina, Cabras, Schettini and Tárraga2012; Rempfer, Brown, & Hamera, Reference Rempfer, Brown and Hamera2011; Uprichard, Kupship, Pine, & Fletcher, 2009; Watzke, Brieger, Kuss, Schöttke, & Wiedl, Reference Watzke, Brieger, Kuss, Schöttke and Wiedl2008).

Dynamic testing is an umbrella term for procedures that examine the effects of brief training on test performance where pre- to post-test change reflects a patient’s learning potential (Grigorenko & Sternberg, Reference Grigorenko and Sternberg1998). It is proposed that dynamic tests provide unique information about a person’s abilities beyond the information that is provided by conventional tests (Caffrey, Fuchs, & Fuchs, Reference Caffrey, Fuchs and Fuchs2008; Grigorenko, 2009). Conventional tests provide information about cognitive deficits that may hamper or facilitate learning, whereas dynamic tests more specifically evaluate patients’ potential to learn and improve cognitive performance. In patients with cognitive impairments due to ABI or a psychiatric or neurodegenerative disorder, several one-session dynamic tests have been applied that target cognitive abilities. The most frequently used is the dynamic Wisconsin Card Sorting Test (dWCST) (Kurtz, Jeffrey, & Rose, Reference Kurtz, Jeffrey and Rose2010; Rempfer et al., Reference Rempfer, Brown and Hamera2011; Uprichard et al., Reference Uprichard, Kupshik, Pine and Fletcher2009; Vaskinn et al., Reference Vaskinn, Sundet, Friis, Ueland, Simonsen, Birkenaes and Andreassen2009; Watzke, Brieger, & Wiedl, Reference Watzke, Brieger and Wiedl2009). The WCST is a renowned measure of executive functioning that can be used to evaluate a patient’s ability to form, maintain, and shift cognitive sets (Heaton, Reference Heaton1981). In contrast to the conventional WCST, the dWCST includes a brief training phase in which additional feedback and instructions are given about the sorting principle after the initial testing phase. The training is followed by a second test phase (i.e., test-train-test design). The degree of change in test performance in response to the training is proposed to represent a patient’s learning potential or capacity to benefit from training.

The dWCST has already shown some evidence of validity in several longitudinal studies that included patients with ABI (Uprichard et al., Reference Uprichard, Kupshik, Pine and Fletcher2009) and psychiatric disorders (Kurtz & Wexler, Reference Kurtz and Wexler2006; Rempfer et al., Reference Rempfer, Brown and Hamera2011; Sergi, Kern, Mintz, & Green, Reference Sergi, Kern, Mintz and Green2005; Watzke et al., Reference Watzke, Brieger, Kuss, Schöttke and Wiedl2008, Reference Watzke, Brieger and Wiedl2009). These studies found that dWCST learning potential was a significant predictor of rehabilitation outcome and provided unique information beyond the conventional WCST in predicting outcome (Sergi et al., Reference Sergi, Kern, Mintz and Green2005). Although promising, there are some issues that affect interpretation of these findings and hence the potential utility of the dWCST in clinical practice. First, adding a training phase may have consequences for the validity of the post-test. For instance, Wiedl, Schöttke, Green, and Nuechterlein (Reference Wiedl, Schöttke, Green and Nuechterlein2004) administered the dWCST to patients with schizophrenia and reported that the training phase alters the cognitive abilities assessed by the post-test. Not surprisingly, both the pre- and post-test scores were associated with tests of executive functioning. The post-test score was, however, additionally associated with verbal memory tests. This suggest that the dWCST pre- and post-test scores reflect different cognitive constructs. These findings have not yet been confirmed by other researchers.

A second issue is that the role of practice effects on pre- to post-test improvement has not yet been examined. Improvements in test performance due to repeated exposure to the same test materials reflect practice effects, whereas pre- to post-test gains due to the brief training reflects learning potential. Just like the conventional WCST, the dWCST has a single solution and is therefore prone to practice effects (Lezak, Howieson, & Loring, Reference Lezak, Howieson and Loring2004). Performance basically depends on discovery of the sort and shift principle. After the sorting principle is discovered or explained, patients are likely to improve their test performance during a second administration of the test (Lezak et al., Reference Lezak, Howieson and Loring2004).

Further psychometric validation of the dWCST would provide researchers and clinicians with a better understanding of what the dWCST actually measures and how to interpret test results. Such research may also guide any necessary refinement of the dWCST for clinical use. Hence, the purpose of this study was to investigate the validity of the dWCST in patients with ABI by: (1) Examining the feasibility of administration and general psychometric properties of the dWCST in terms of score distributions, administration time, floor and ceiling effects, intercorrelations between different learning potential indices (i.e., post-test score, gain score), and associations between the post-test score and gain score and other neuropsychological tests. (2) Determining whether pre- to post-test improvement reflects learning potential or practice effects. (3) Determining whether the dWCST can discriminate between patients and controls by comparing dWCST performance of patients with ABI and healthy controls matched on age, gender, and education.

Hypotheses

We hypothesized that the distribution of the dWCST learning potential indices would demonstrate an absence of floor or ceiling effects and that the learning potential indices would be highly intercorrelated. We expected that the dWCST scores would be significantly associated with other measures of executive functioning at rehabilitation admission. Also, we hypothesized that patients and controls, would demonstrate a significant change in pre- to post-test scores for the dWCST, whereas patients who received a repeated assessment of the WCST (rWCST) would demonstrate no significant pre- to post-test change. Compared to the rWCST group, patients in the dWCST group would demonstrate similar pre-test scores, but significantly higher gain and post-test scores. Furthermore, we expected that, compared to healthy controls, patients would demonstrate significantly lower pre-test, post-test, and gain scores for the dWCST.

Methods

Participants

Patients were recruited from five rehabilitation centers in The Netherlands. For all patients (i.e., the dWCST and rWCST group), the same in- and exclusion criteria were used. Inclusion criteria were (1) diagnosis of traumatic or non-traumatic ABI based on medical records; (2) ≥18 years of age; (3) sufficient command of the Dutch language based on clinical judgment; and (4) completion of the dWCST at admission to rehabilitation. Exclusion criteria were: (1) severe aphasia based on a Dutch Aphasia Foundation (Stichting Afasie Nederland, SAN; Deelman, Koning-Haanstra, Liebrand, & van de Burg, Reference Deelman, Koning-Haanstra, Liebrand and van de Burg1987) scale score less than 4 or clinical judgment; (2) premorbid psychiatric disorder and/or substance abuse for which hospital admission was necessary; (3) minimally conscious state or post-traumatic amnesia at the time of assessment; (4) degenerative or progressive brain disease; (5) active participation in another study to avoid participation burden; (6) no informed consent; and (7) completion of the conventional WCST during a neuropsychological assessment.

The control group consisted of volunteers who were recruited through personal contacts. Inclusion criteria were: (1) ≥18 years of age; (2) sufficient command of the Dutch language based on clinical judgment; and (3) written informed consent. Exclusion criteria were: (1) history of a neurological event or psychiatric disorder; and (2) cognitive impairments, indicated by a Montreal Cognitive Assessment (MoCA; Nasreddine et al., Reference Nasreddine, Phillips, Bedirian, Charbonneau, Whitehead, Collin and Chertkow2005) score below 24. The medical ethics committee of the University Medical Center Utrecht and the five participating rehabilitation centers approved the study protocol. All data were obtained in compliance with regulations of the participating rehabilitation centers. All participants gave informed consent.

Feasibility of administration and general psychometric properties

Data from patients who were participating in a longitudinal cohort study examining factors influencing outcome of ABI rehabilitation (dWCST group) were examined. These patients were recruited from November 2012 to December 2013 from inpatient clinics of five rehabilitation centers in The Netherlands: Adelante Zorggroep, Hoensbroek; De Hoogstraat Rehabilitation, Utrecht; Reade Rehabilitation Center, Amsterdam; Rijndam Rehabilitation, Rotterdam; and Rehabilitation Center Tolbrug, ‘s-Hertogenbosch.

Learning potential versus practice effects

A subgroup of patients from the dWCST group was selected, namely all patients who were recruited at De Hoogstraat Rehabilitation (dWCST subgroup). We only included patients from De Hoogstraat Rehabilitation to ensure uniformity of administration procedures from staff at the same clinic, and also achieve a large sample size for the main experimental condition. dWCST performance of these patients was compared to another group of patients (rWCST group). Patients in the rWCST group were recruited from July 2013 to December 2013 from the inpatient clinic of De Hoogstraat Rehabilitation. These patients were not eligible to participate in the dWCST group because they were admitted to one of the inpatient clinics in which the recruitment period had already ended.

Patients with ABI Versus Healthy Controls

dWCST performance of the dWCST subgroup was compared to a group of healthy controls (control group) matched on age, gender, and education. The control group was recruited from October 2013 to December 2013 as a reference sample to compare performance of patients and healthy controls.

Measures

Wisconsin Card Sorting Test (WCST)

The participant was given two decks of 64 cards containing figures which differ in color (yellow, red, green, and blue), shape (star, square, circle, and cross), and number (one, two, three, and four). The participant was instructed to place each card below one of four stimulus cards. The participant had to deduce the sorting principle from the experimenter’s feedback (“right”, “wrong”) (Heaton, Reference Heaton1981). The administration time of the conventional WCST is approximately 20 to 30 min.

Cognitive functioning

A nationally recommended core battery of cognitive tests (Nederlands Instituut van Psychologen, 2010) was administered by a psychological assistant. Verbal memory was assessed with the Rey Auditory Verbal Learning Test (RAVLT; Rey, Reference Rey1958). Attention was evaluated with the Trail Making Test parts A and B (TMT; Reitan, Reference Reitan1956). Part A is a measure of psychomotor speed and Part B is a measure of divided attention and executive functioning. Executive functioning was also evaluated with the letter fluency test (LFT; Schmand, Groenink, & van den Dungen, Reference Schmand, Groenink and Van den Dungen2008). Language performance was evaluated with the Boston Naming Test (Kaplan, Goodglass, & Weintraub, Reference Kaplan, Goodglass and Weintraub1983) short form and the Category Fluency Test (CFT; Van der Elst, van Boxtel, Van Breukelen, & Jolles, Reference Van der Elst, Van Boxtel, Van Breukelen and Jolles2006). The Star Cancellation Test (SCT; Wilson, Cockburn, & Halligan, Reference Wilson, Cockburn and Halligan1987) was used to assess patients’ visual perception and to screen for unilateral spatial neglect. In addition, premorbid intelligence was estimated with the Dutch version of the National Adult Reading Test (NART; Nelson, Reference Nelson1982), the ‘Nederlandse Leestest voor Volwassenen’ (NLV; Schmand, Lindeboom, & van Harskamp, Reference Schmand, Lindeboom and Van Harskamp1992). These tests are described in more detail elsewhere (Lezak et al., Reference Lezak, Howieson and Loring2004).

Procedures

For all patients, the treating rehabilitation physician confirmed the eligibility criteria and obtained informed consent. After patients provided written informed consent, demographic and disease characteristics were obtained from their medical records. In the dWCST group (and thus the dWCST subgroup), the dWCST was administered within approximately 2 weeks of enrollment. Within approximately the same week, a cognitive screening was conducted by a psychological assistant as part of routine assessment procedures. In the rWCST group, the rWCST was administered within 1 week after informed consent was obtained. In the healthy control group, a home visit was scheduled. During the home visit, participants were checked for eligibility. If eligible, the dWCST was administered.

Administration procedures of the dWCST and rWCST

The dWCST and rWCST were administered in a quiet room by a trained clinician or trained neuropsychology student. Before commencement, patients and healthy controls were asked to give a detailed description of the four stimulus cards (one red triangle, two green stars, three yellow crosses, four blue circles) to assess potential visual (e.g., achromatopsia) or cognitive (e.g., visual form agnosia) problems that could influence test performance. The test was not completed when such problems were considered to significantly influence performance.

During administration of the dWCST, a pre-test – train – post-test paradigm was used. The dWCST was administered according to the test protocol of Wiedl and Wienöbst (Reference Wiedl and Wienöbst1999) which was translated from German into Dutch. The dWCST pre- and post-test were administered according to standard WCST administration procedures (Heaton, Reference Heaton1981). The first deck of 64 cards was used for the pre-test as well as for the training, and the second deck of 64 cards was used for the post-test. The training phase consisted of three major elements: (1) explanation of the sorting rules (e.g., “There are three possible ways to match the cards: you can match the card by color, by number of the objects, or by shape”); (2) during the card sorting, the patient was told why a response was right or wrong after each card sort (e.g., “This was wrong, we don’t sort for color”); and (3) how many consecutive correct responses the patient needed for the rule to change (e.g., “After ten consecutive correct sorts, the rule will change. You will then sort for color or number”).

During administration of the rWCST, a test-test-test procedure was adopted. The deck of 64 cards was administered three times using standard WCST procedures (Heaton, Reference Heaton1981). The first deck of 64 cards was used for the first two administrations (i.e., pre-test and test without training) and the second deck of 64 cards was used for the third administration (i.e., post-test).

For both the dWCST and rWCST, the total number of correct responses was recorded for the pre-test and post-test. For both test phases, the score ranged from 0 to 64 with higher scores indicating better performance. The time needed to administer the dWCST and rWCST was recorded.

Statistical analysis

dWCST indices

For all patients and controls, the pre-test score and three different learning potential indices were calculated. The computational methods are described below. The learning potential indices were selected based on their use in prior studies to index learning potential (Fiszdon & Johannesen, Reference Fiszdon and Johannesen2010; Weingartz, Wiedl, & Watzke, Reference Weingartz, Wiedl and Watzke2008) and their feasibility for use in clinical settings. For all indices, higher scores indicate better performance.

Pre-test score

Total correct after the pre-test.

Post-test score

Total correct after the training phase.

Gain score

Ratio calculated by dividing actual performance change (i.e., difference score) by potential performance change. The following formula was used: (post-test total correct – pre-test total correct) / (64 – pre-test total correct) in which 64 is represents the maximum total correct that can be achieved on the post-test (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008).

Groups

Previously established cutoff values were used based on the reliable change approach by Wiedl et al. (Reference Wiedl and Wienöbst1999). Patients were categorized as “high achiever” (pre- and post-test ≥43 correct); “strong learner” (pre- to post-test improvement ≥15 points); “poor learner” (pre- to post-test improvement <15 points); or “decliner” (pre- to post-test decline ≥15 points). The Reliable Change Generator (Devilly, Reference Devilly2005) was used to determine whether these cutoff values were reliable (Jacobson & Truax, Reference Jacobson and Truax1991) in our sample of patients with ABI. Based on the rWCST pre-test standard deviation (SD=12.1) and test-retest reliability (rpre-test, post-test=0.81) a reliable change index of 15 was calculated. Thus, reliable change was considered present when the pre- and post-test differed by ≥15 points which is in agreement with the previously used cutoff values.

Feasibility of administration and general psychometric properties

The distribution of numerical learning potential scores was evaluated using Kolmogorov Smirnov, skewness, and kurtosis tests. In case of a non-normal distribution, transformations were performed to improve normality. If these did not improve normality, non-parametric statistics were used. Feasibility of the dWCST was investigated by calculating the time needed to administer the dWCST. Floor and ceiling effects for the dWCST pre-test and numerical learning potential indices were examined. Floor or ceiling effects were considered present if more than 15% of all patients obtained the lowest (post-test score=0; gain score=−64) or highest (post-test score=64; gain score=1) possible score (Terwee et al., Reference Terwee, Bot, De Boer, Van der Windt, Knol, Dekker and De Vet2007). Pearson or Spearman correlations were used to examine intercorrelations between the post-test score and gain score and their association with other neuropsychological tests. Associations between the dWCST pre- and post-test scores and between the pre-test score and other neuropsychological tests were also examined for comparison purposes. Between-group differences in neuropsychological test performance was evaluated using One-way between-groups analyses of variance and independent samples t tests or Kruskal-Wallis and Mann-Whitney U Tests.

Learning potential versus practice effects

First, independent samples t tests or Mann-Whitney U Tests, and Chi square tests were used to examine differences between the dWCST subgroup and the rWCST group regarding age, gender, education, and functional independence. Between-group differences in dWCST and rWCST scores (pre-test score, post-test score, gain score) then were examined using independent samples t tests or Mann-Whitney U Tests. For both groups, paired samples t tests or Wilcoxon Signed-Rank tests were used to determine whether there was a significant change in pre- to post-test performance. Also, between-group differences in the number of poor learners and strong learners were examined using Chi square tests. Decliners and high achievers were not included in this analysis. A decline in performance indicates neither learning potential nor a practice effect, and high achievers do not require a training phase to perform adequately.

Patients with ABI versus healthy controls

First, independent samples t tests or Mann-Whitney U Tests, and χ2 were used to examine whether the dWCST subgroup and the control group were adequately matched for age, gender, and education. Second, differences in dWCST scores (pre-test score, post-test score, and gain score) between the dWCST subgroup and the control group were evaluated using independent samples t tests or Mann-Whitney U Tests. For both groups, paired samples t tests or Wilcoxon Signed-Rank tests were used to determine whether there was a significant change in pre- to post-test performance. Furthermore, between-group differences in the number of patients with low performance (decliners and poor learners) versus adequate performance (strong learners and high achievers) between the dWCST subgroup and the control group were examined using χ2 tests.

For all analyses, Pearson or Spearman correlation coefficients between 0.30 and 0.49 were considered to be moderate and correlations exceeding 0.50 were interpreted as large (Cohen, Reference Cohen1983). Due to the large number of associations examined, alpha was set at 0.01. Data were analyzed using SPSS version 21.0.

Results

Participants

For the dWCST group, 125 patients with ABI were recruited from the five rehabilitation centers (response rate 78.1%). Of these, 21 patients were excluded for the following reasons: the conventional WCST was already administered during the routine neuropsychological assessment (n=3); refusal to complete the dWCST (n=2); withdrew from study (n=4); dWCST was not administered due to early discharge (n=4); dWCST was discontinued by the examiner because the test was perceived as too demanding for the patient (n=7). A total of 104 patients were included in the dWCST group. Of these, 63 patients were recruited at De Hoogstraat Rehabilitation and thus were selected for the dWCST subgroup. For the rWCST group, 32 patients were recruited. Of these, the rWCST was discontinued by three patients during or after the pre-test, and for one patient the rWCST was discontinued by the examiner because the test was perceived as too demanding for the patient, leaving a total of 28 patients. Based on clinical judgment, all 28 patients in the rWCST group and all 104 patients in the dWCST group, showed sufficient visual and cognitive abilities to complete the test. For the control group, 30 healthy controls were recruited. The MoCA scores of the control participants ranged from 24 to 30 (mean, 27.6±1.73), indicating adequate cognitive functioning for all healthy controls.

There were no differences between patients in the dWCST subgroup and the rWCST group regarding age (U=676.0; z=−1.77; p=.08; r=−.19), gender (χ2=1.90; p=.17; phi=−.14), education (χ2=.68; p=.41; phi=−.09), and functional independence (χ2=2.16; p=.34; Cramer’s V=.16). The dWCST subgroup was adequately matched with the control group for gender (χ2=.27; p=.61; phi=.05), age (U=897.0; z=−.40; p=.69; r=−.04), and education (χ2=.63; p=.43; phi=.08). On average, the dWCST was administered significantly later than the rWCST (median, 27 vs. 11 days after admission, respectively; U=435.5; z=−3.66; p<.001; r=.39). Table 1 shows the characteristics for all participants groups.

Table 1 Characteristics of the participant groups

a High education ≥ higher vocational education.

b n=95.

c n=27.

Score Distributions and Administration Time

All learning potential indices showed a left-skewed, non-normal distribution (skewness=−.98 to −.99). Data transformations did not improve normality. Therefore, non-parametric statistics were used. dWCST administration time varied from 10 to 55 minutes (median=30.0 min). In total, 86.5% of patients completed the dWCST in 40 minutes or less. In Table 2, an overview of the dWCST scores is presented. When classifying patients into groups, 34.9% were high achievers, 42.9% were strong learners, and 22.2% were poor learners. There were no decliners. Figure 1 shows the median dWCST scores for the pre-test, training and post-test for the three learner groups.

Fig. 1 Median dynamic Wisconsin Card Sorting Test scores for the patient subgroup divided into poor learners, strong learners, and high achievers.

Table 2 Descriptives, floor- and ceiling effects, and intercorrelations of dynamic Wisconsin Card Sorting Test scores of the dWCST group

*p≤.01

a Gain score=(post-test total correct – pre-test total correct)/(64 – pre-test total correct); dWCST=dynamic Wisconsin Card Sorting Test; IQR=interquartile range

Floor and Ceiling Effects and Intercorrelations

There were no floor or ceiling effects for the dWCST post-test score and gain score. In total, 11.1% of patients had ≥58 items correct and 1.0% had all 64 post-test items correct. The post-test score showed a large, significant association with the gain score (r=.62; p<.001).

Associations with Neuropsychological Tests

In Table 3, the correlation coefficients are shown for the associations between the dWCST scores and the neuropsychological tests. The pre-test score correlations were presented for comparison. The pre-test score was not significantly associated with any of the cognitive tests. The post-test score and gain score showed significant, moderate associations with long-term memory (rpost-test=37; p<.001; rgain score=.31; p=.001) and recognition memory (rpost-test=.37; p<.001; rgain score=.34; p=.001). Furthermore, the post-test score showed a moderate significant association with short-term memory (r=.32; p=.001) and a small, significant association with language (r=.26; p=.01; r=.26; p=.009). The dWCST pre- and post-test score showed a large, significant association (r=.50; p<.001).

Table 3 Spearman correlations between the dynamic Wisconsin Card Sorting Test pre-test, post-test score, and gain score, and Neuropsychological Tests of the dWCST group

**p≤0.01; * p≤0.05.

a Some patients could not complete all tests due to for instance reading or visual problems (N range 97–104);

b R-AVLT=Rey Auditory Verbal Learning Test; WCST-64=Wisconsin Card Sorting Test-64 (Kongs, Thompson, Iverson, & Heaton, Reference Kongs, Thompson, Iverson and Heaton2000).

c Higher scores reflect worse performance.

Between-group differences in neuropsychological test performance are presented in Table 4. No significant between-group differences in neuropsychological test performance were observed at a 0.01 level. For two tests, there was a trend toward significance. At a 0.05 level of significance, language scores were significantly lower for poor learners compared to high achievers (U=320.0; z=−2.5; p=.01; r=−0.31), and poor and strong learners obtained lower scores for the measure of attention and executive functioning compared to high achievers (Upoor=256.5; z=−2.4; p=0.02; r=−0.31; Ustrong=437.5; z=−2.0; p=.04; r=−0.24) (Table 4).

Learning Potential Versus Practice Effects

Test scores of the dWCST subgroup were compared to test scores of the rWCST group. The median scores for both patient groups are presented in Table 5. There were no significant between-group differences in pre-test performance (U=882.0; z=.000; p=1.00). Compared to patients who were administered the dWCST, the rWCST group had lower scores for the post-test (U=475.5; z=−3.50; p<.001; r=.37), and gain score (U=416.0; z=−4.01; p<.001; r=−.42). Furthermore, no significant change from pre- to post-test scores was found in the rWCST group (Z=−1.3; p=.182; r=−.17). In the dWCST group, the pre- to post-test improvement was significant (Z=−6.10; p<.001; r=−.54). There were significantly more strong learners in the dWCST group compared to the rWCST group (χ2=17.3; p<.001; phi=−.55; 42.9% vs. 3.6%, respectively). In Figure 2, the median dWCST scores are displayed for the dWCST and rWCST group as well as the control group.

Fig. 2 Median score for dynamic Wisconsin Card Sorting Test subgroup, repeated Wisconsin Card Sorting Test group, and the control group (dynamic Wisconsin Card Sorting Test).

Table 4 Between-group differences in neuropsychological test performance

**p≤.01; *p≤.05.

a Some patients could not complete all tests due to for instance reading or visual problems.

b R-AVLT=Rey Auditory Verbal Learning Test.

c Higher scores reflect worse performance.

Patients with ABI Versus Healthy Controls

Test scores of the dWCST subgroup were compared to the control group. The median dWCST scores for both groups are presented in Table 5. Compared to the dWCST subgroup, the control group had significantly higher scores for the pre-test (U=.599.0; z=−2.85; p=.004; r=−.30), and post-test (U=561.5; z=−3.16; p=.002; r=−.33) (see Figure 2). There were no significant between-group differences for the gain score (U=887.5; z=−.47; p=.636; r=−.05). Just like in the dWCST subgroup, the control group showed significant pre- to post-test improvement (Z=−3.9; p<.001; r=−.50). Furthermore, the difference in the number of poor learners between the dWCST subgroup and the control group was non-significant, but showed a trend toward significance (χ2=5.22; p=.02; phi=.24; 65.1% vs. 40.0%, respectively).

Table 5 Between-group comparisons of dynamic and repeated Wisconsin Card Sorting Test scores

**p≤0.01, *p≤0.05.

a dWCST=dynamic Wisconsin Card Sorting Test (test-train-test); rWCST=repeated Wisconsin Card Sorting Test (test-test-test).

b Learning potential group (poor learners vs. strong learners)×participant group (dWCST subgroup, rWCST group).

c Learning potential group (poor learners vs. strong learners/high achievers)×participant group (dWCST subgroup, control group).

Discussion

This study investigated the validity of the dWCST for assessing learning potential in patients with ABI. The results showed that adding training to the WCST takes, on average, only approximately 10 min additional administration time. The dWCST showed no floor or ceiling effects, and a large intercorrelation between the two numerical learning potential indices. The pre-test score showed no significant associations with other neuropsychological tests, whereas the numerical learning potential indices were significantly associated with language and/or memory. The dWCST group had a similar pre-test score to that of the rWCST group, but a higher post-test and gain score. This is evidence that the dWCST measures learning potential opposed to practice effects. Compared to healthy controls, patients obtained similar gains but significantly lower pre- and post-test scores. The ratio of poor learners between-groups was not significantly different. This is partial support for the sensitivity of the dWCST to discriminate patients from controls.

Feasibility of Administration and General Psychometric Properties

The dWCST showed good feasibility and adequate psychometric properties. As hypothesized, the learning potential indices on the dWCST were significantly associated with one another which is in line with previous findings (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). This indicates that the learning potential indices measure a similar construct. The pattern of associations between the post-test score and gain score and memory performance is consistent with a study by Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004), thus reinforcing the view that memory is associated with learning (Boosman, Visser-Meily, Winkens, & Van Heugten, Reference Boosman, Visser-Meily, Winkens and van Heugten2013; Lezak et al., Reference Lezak, Howieson and Loring2004). Effective performance requires patients to remember what they have learned during the brief training. Post-test scores were also significantly associated with language tests, which may be due to the verbal nature of the training. In contrast with our hypothesis, and the findings of Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004), the post-test was not significantly associated with tests of executive functioning. This may be explained by the use of different tests of executive functioning between the studies. Dissociations in performance among executive tests and low between-test correlations have been reported previously (Jurado & Rosselli, Reference Jurado and Rosselli2007). In the current study, the LFT and TMT were used as measures of executive functioning. Similar to the WCST, these tests measure inhibition (Jurado & Rosselli, Reference Jurado and Rosselli2007). However, the WCST measures several additional functions (i.e., rule detection, concept formation, set maintenance) (Jurado & Rosselli, Reference Jurado and Rosselli2007) which may explain the lack of a significant association between the tests. Wiedl et al. (Reference Wiedl, Schöttke, Green and Nuechterlein2004) used the Tower of Hanoi which requires patients to formulate and execute a strategy while complying with a set of rules, and thus may assess abilities more closely related to the dWCST.

The substantial memory component and verbal nature of the training may also explain why some patients with low pre-test performance showed a substantial improvement after training whereas others showed only a marginal shift in performance. In particular, it is possible that patients who have considerable verbal memory and/or language impairments, may benefit less from training on this particular task compared to patients with better language and verbal memory functioning. Although other authors have suggested distinct neuro-cognitive profiles between the three learner groups in psychiatric populations (Bisoglio, Mervis, & Choi, Reference Bisoglio, Mervis and Choi2014; Kurtz et al., Reference Kurtz and Wexler2006; Rempfer, Hamera, Brown, & Bothwell, Reference Rempfer, Hamera, Brown and Bothwell2006; Wiedl, Wienöbst, Schöttke, Green, & Nuechterlein, Reference Wiedl, Wienöbst, Schöttke, Green and Nuechterlein2001;), between-group differences in this study were only found at trend-level. Also, the learning potential indices were not significantly associated with demographic characteristics, level of functional independence, or diagnosis (traumatic vs. non-traumatic ABI). It is possible that a range of other injury-related, cognitive or psychological factors contributed to the observed differences in amount of learning (e.g., injury location, visual memory, motivation) (Boosman et al., Reference Boosman, Visser-Meily, Winkens and van Heugten2013).

Learning Potential Versus Practice Effects

The lack of significant pre- to post-test improvement on the rWCST suggests that training effects were apparent over and above practice effects. Although only one patient in the rWCST group showed significant gains (i.e., strong learner), several other patients did show some improvement in pre- to post-test performance (see Figure 2). These marginal gains may reflect test-specific improvement rather than learning potential. For the conventional WCST, improvements between the first and second deck of cards have been attributed to continued exposure and corrective feedback (Merrick, Donders, & Wiersum, Reference Merrick, Donders and Wiersum2003). Differentiating test-specific gains from learning gains due to the training is a complex process. The lack of improvement on the rWCST may also be partly attributed to the demanding nature of the repeated administration condition in terms of the high number of trials and limited instructions and feedback provided by the experimenter. This may have influenced patients’ motivation and attention during the task. These issues may have contributed to the significant decline in performance observed for one patient in the rWCST group. Such a decline in performance has also been observed between the first and second deck of the conventional WCST (Sherer, Nick, Millis, & Novack, Reference Sherer, Nick, Millis and Novack2003).

Patients with ABI Versus Healthy Controls

In line with the results of a previous study that compared dWCST performance of healthy controls and patients with schizophrenia (Ohrmann et al., Reference Ohrmann, Kugel, Bauer, Siegmund, Kölkebeck, Suslow and Pedersen2008), patients obtained similar gains but lower pre- and post-test scores compared to healthy controls. This provides partial support for the sensitivity of the dWCST to discriminate patients from controls. The finding that patients and controls obtained similar gain scores may be explained by the fact that gain scores can produce disproportionately high or low scores (Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). Of note is that additional analyses revealed no significant differences in dWCST scores of healthy controls and patients classified as high achievers. This, along with the finding that there were no decliners, suggests that the dWCST has greatest utility for differentiating between poor and strong learners.

Learning Potential Indices

The learning potential indices can be used to interpret individual test performance. In this study, three different learning potential indices were used: the post-test score, gain score and group classification. Several studies have discussed the strengths and limitations of these and other indices in more detail (Fiszdon & Johannesen, Reference Fiszdon and Johannesen2010; Waldorf, Wiedl, & Schöttke, Reference Waldorf, Wiedl and Schöttke2009; Weingartz et al., Reference Weingartz, Wiedl and Watzke2008). Most importantly, the learning potential indices need to be interpreted together to provide a clearer picture of a patient’s learning profile, that is, a patient’s initial performance, magnitude of change from training and their post-test performance. These scores can also be used to examine within-group variability when using the categorical approach. Any of the learning potential indices viewed in isolation could be misleading.

Limitations

Several limitations of this study should be noted. First, patients were not randomly allocated to the two assessment procedures (dWCST; rWCST). Although the groups did not differ in their level of functional independence and demographic characteristics, it is conceivable that between-group differences were confounded by other baseline differences between the two groups (e.g., cognition) and differences in the timing of assessment after admission to rehabilitation. The dWCST was administered significantly later than the rWCST. In the rWCST group, there were seven patients who were assessed in an early phase after ABI (<3 weeks post-ABI) which may have negatively influenced their potential for gains. However, additional analyses, including only patients who were assessed at least 3 weeks post-ABI revealed a similar pattern of non-significant (i.e., pre-test score) and significant (i.e., post-test score and gain score) findings.

A second limitation is the mixed etiology of our sample and the lack of injury specific analyses. However, clinical practice is also mixed. Third, although this study provides support for the validity of the dWCST, the clinical value of the dWCST for predicting rehabilitation outcomes was not established. Future studies could, for instance, examine the neuro-cognitive and psychological characteristics and long-term outcomes of poor and strong learners.

Conclusion

This study provides preliminary support for the validity of the dWCST as an instrument for assessing learning potential in patients with ABI. Evidence is provided for the absence of substantial practice effects as well as its ability to discriminate between patients with ABI and healthy controls. Further research is needed to investigate the predictive validity of the dWCST in comparison to other indices that are commonly used for prognostic purposes (e.g., functional independence).

Acknowledgments

This study is funded by the National Initiative Brain and Cognition (NIBC). This quick-result project is embedded in the pillar “The Healthy Brain, Program Cognitive Rehabilitation” [grant number 056-11-014]. The authors thank the participating rehabilitation centers: Adelante Zorggroep, Hoensbroek; De Hoogstraat Rehabilitation, Utrecht; Reade, Rehabilitation Center, Amsterdam; Rijndam Rehabilitation, Rotterdam; Rehabilitation Center Tolbrug, ’s-Hertogenbosch. Also, thanks to Ilse Blaas, Femke Coenen, Nelleke Kip, Elvire Mastboom, Linette Thiessens, Maartje van der Blij, Marel Vermeulen, and Syanah Wynn for their assistance in data collection. There were no competing interests.

References

Bisoglio, J., Mervis, J., & Choi, J. (2014). Meta-analysis of “learning potential” on the dynamic Wisconsin Card Sorting Test (D-WCST): Distinct cognitive subgroups with divergent functional outcomes. Abstracts of the 4th Biennial Schizophrenia International Research Conference. Schizophrenia Research, 153, S156.Google Scholar
Boosman, H., Visser-Meily, J. M. A., Winkens, I., & van Heugten, C. M. (2013). Clinicians’ views on learning in brain injury rehabilitation. Brain Injury, 27, 685688. doi: 10.3109/02699052.2013.775504 Google Scholar
Caffrey, E., Fuchs, D., & Fuchs, L. S. (2008). The predictive validity of dynamic assessment. A review. The Journal of Special Education, 41, 254270. doi: 10.1177/0022466907310366 Google Scholar
Calero, M. D., & Navarro, E. (2004). Relationship between plasticity, mild cognitive impairment and cognitive decline. Archives of Clinical Neuropsychology, 19, 653660. doi: 10.1016/j.acn.2003.08.008 Google Scholar
Cohen, J. W. (1983). Statistical power analysis for the behavioral sciences, second edition. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Deelman, B. G., Koning-Haanstra, M., Liebrand, W. G. B., & van de Burg, W. (1987). Handleiding van de SAN test. Lisse: Swets en Zeitlinger.Google Scholar
Devilly, G. J. (2005). ClinTools Software for Windows (Version 3.5) [Computer program]. Melbourne, Australia: Psytek Ltd.Google Scholar
Fernández-Ballesteros, R., Botella, J., Zamarrón, M. D., Molina, M. A., Cabras, E., Schettini, R., & Tárraga, L. (2012). Cognitive plasticity in normal and pathological aging. Clinical Interventions in Aging, 7, 1525. doi: 10.2147/CIA.S27008 CrossRefGoogle ScholarPubMed
Fiszdon, J. M., & Johannesen, J. K. (2010). Comparison of computational methods for the evaluation of learning potential in schizophrenia. Journal of the International Neuropsychological Society, 16, 613620. doi: 10.1017/S1355617710000317 Google Scholar
Grigorenko, E. L., & Sternberg, R. J. (1998). Dynamic testing. Psychological Bulletin, 124, 75111.CrossRefGoogle Scholar
Grigorenko, E. L. (2009). Dynamic assessment and response to intervention: two sides of one coin. Journal of Learning Disabilities, 42, 111132. doi: 10.1177/0022219408326207 CrossRefGoogle ScholarPubMed
Heaton, R. K. (1981). Wisconsin Card Sorting Test (WCST). Odessa, FL: Psychological Assessment Resources.Google Scholar
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 1219. doi: 10.1037/0022-006X.59.1.12 Google Scholar
Jurado, M. B., & Rosselli, M. (2007). The elusive nature of executive functions: A review of our current understanding. Neuropsychology Review, 17, 213233. doi: 10.1007/s11065-007-9040-z Google Scholar
Kaplan, E., Goodglass, H., & Weintraub, S. (1983). Boston naming test. Philadelphia: Lea & Febiger.Google Scholar
Kongs, S. K., Thompson, L. L., Iverson, G. L., & Heaton, R. K. (2000). Wisconsin Card Sorting Test – 64 card version professional manual. Lutz, FL: Psychological Assessment Resources.Google Scholar
Kurtz, M. M., & Wexler, B. E. (2006). Differences in performance and learning proficiency on the Wisconsin Card Sorting Test in schizophrenia: Do they reflect distinct neurocognitive subtypes with distinct functional profiles? Schizophrenia Research, 81, 167171. doi: 10.1016/j.schres.2005.09.003 Google Scholar
Kurtz, M. M., Jeffrey, S. B., & Rose, J. (2010). Elementary neurocognitive function, learning potential and everyday life skills in schizophrenia: What is their relationship? Schizophrenia Research, 116, 280288. doi: 10.1016/j.schres.2009.08.011 Google Scholar
Lezak, M., Howieson, D., & Loring, D. (2004). Neuropsychological assessment. 4th ed. New York: Oxford University Press.Google Scholar
Merrick, E. E., Donders, J., & Wiersum, M. (2003). Validity of the WCST-64 after traumatic brain injury. The Clinical Neuropsychologist, 17, 153158. doi: 10.1076/clin.17.2.153.16521 CrossRefGoogle ScholarPubMed
Nasreddine, Z. S., Phillips, N. A., Bedirian, V., Charbonneau, S., Whitehead, V., Collin, I., … Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53, 695699. doi:10.1111/j.1532 Google Scholar
Nelson, H. E. (1982). The National Adult Reading Test (NART): Test manual. Windsor: NFER-Nelson.Google Scholar
Nederlands Instituut van Psychologen sectie Revalidatie en sectie Neuropsychologie (2010). Richtlijn voor kort NPO bij patiënten met een beroerte. Augustus 2010.Google Scholar
Ohrmann, P., Kugel, H., Bauer, J., Siegmund, A., Kölkebeck, K., Suslow, T., … Pedersen, A. (2008). Learning potential on the WCST in schizophrenia is related to the neuronal integrity of the anterior cingulated cortex as measured by proton magnetic resonance spectroscopy. Schizophrenia Research, 106, 156163. doi: 10.1016/j.schres.2008.08.005 Google Scholar
Reitan, R. (1956). Trail Making Test: Manual for administration, scoring and interpretation. Bloomington, IN: Indiana University.Google Scholar
Rempfer, M., Hamera, E., Brown, C., & Bothwell, R. J. (2006). Learning proficiency on the Wisconsin Card Sorting Test in people with serious mental illness: What are the cognitive characteristics of good learners? Schizophrenia Research, 87, 316322. doi: 10.1016/j.schres.2006.05.012 Google Scholar
Rempfer, M., Brown, C., & Hamera, E. (2011). Learning potential as a predictor of skill acquisition in people with serious mental illness. Psychiatry Research, 185, 293295. doi: 10.1016/j.psychres.2009.12.009 Google Scholar
Rey, A. (1958). L’Examen Clinique en Psychologie. Paris, France: Press Universitaire de France.Google Scholar
Schmand, B., Groenink, S. C., & Van den Dungen, M. (2008). Letter fluency: psychometrische eigenschappen en Nederlandse normen. Tijdschrift voor Gerontology en Geriatrie, 39, 6476.Google Scholar
Schmand, B., Lindeboom, J., & Van Harskamp, F. (1992). Nederlandse Leestest voor volwassenen. Lisse: Swets & Zeitlinger.Google Scholar
Sergi, M. J., Kern, R. S., Mintz, J., & Green, M. F. (2005). Learning potential and the prediction of work skill acquisition in schizophrenia. Schizophrenia Bulletin, 31, 6772. doi: 10.1093/schbul/sbi007 Google Scholar
Sherer, M., Nick, T. G., Millis, S. R., & Novack, T. A. (2003). Use of the WCST and the WCST-64 in the assessment of traumatic brain injury. Journal of Clinical and Experimental Neuropsychology, 25, 512520. doi: 10.1076/jcen.25.4.512.13877 Google Scholar
Terwee, C. B., Bot, S. D. M., De Boer, M. R., Van der Windt, D. A. W. M., Knol, D. L., Dekker, , … De Vet, H. C. W. (2007). Quality criteria were proposed for measurement properties of health status questionnaires. Journal of Clinical Epidemiology, 60, 3442.Google Scholar
Uprichard, S., Kupshik, G., Pine, K., & Fletcher, B. C. (2009). Dynamic assessment of learning ability improves outcome prediction following acquired brain injury. Brain Injury, 23, 278290.Google Scholar
Van der Elst, W., Van Boxtel, M. P., Van Breukelen, G. J., & Jolles, J. (2006). Normative data for the animal, profession and letter M naming verbal fluency tests for Dutch speaking participants and the effects of age, education, and sex. Journal of the International Neuropsychological Society, 12, 8089. doi: 10.1017/S1355617706060115 Google Scholar
Vaskinn, A., Sundet, K., Friis, S., Ueland, R., Simonsen, C., Birkenaes, A. B., … Andreassen, O. A. (2009). Using the Wisconsin Card Sorting Test to assess learning potential in normal IQ schizophrenia: Does it have potential? Nordic Journal of Psychiatry, 63, 405411. doi: 10.1080/08039480902879941 Google Scholar
Wade, D. T. (2005). Describing rehabilitation interventions. Clinical Rehabilitation, 19, 811818. doi: 10.1191/0269215505cr923ed Google Scholar
Waldorf, M., Wiedl, K. H., & Schöttke, H. (2009). The concordance of three reliable change indexes: An analysis applying the dynamic Wisconsin Card Sorting Test. Journal of Cognitive Education and Psychology, 8, 6380. doi: 10.1891/1945-8959.8.1.63 CrossRefGoogle Scholar
Watzke, S., Brieger, P., Kuss, O., Schöttke, H., & Wiedl, K. H. (2008). A longitudinal study of learning potential and rehabilitation outcome in schizophrenia. Psychiatric Services, 59, 248255. doi: 10.1176/appi.ps.59.3.248 Google Scholar
Watzke, S., Brieger, P., & Wiedl, K. H. (2009). Prediction of vocational rehabilitation outcome in schizophrenia: Incremental prognostic validity of learning potential beyond basic cognitive performance. Journal of Cognitive Education and Psychology, 8, 5262. doi: 10.1891/1945-8959.8.1.52 Google Scholar
Weingartz, S., Wiedl, K. H., & Watzke, S. (2008). Dynamic assessment of executive functioning: (How) can we measure change? Journal of Cognitive Education and Psychology, 7, 368387. doi: 10.1891/194589508787724088 Google Scholar
Wiedl, K. H., & Wienöbst, J. (1999). Interindividual differences in cognitive remediation research with schizophrenic patients – indicators of rehabilitation potential? International Journal of Rehabilitation Research, 22, 5559. doi: 10.1097/00004356-199903000-00007 Google Scholar
Wiedl, K. H., Wienöbst, J., Schöttke, H. H., Green, M. F., & Nuechterlein, K. H. (2001). Attentional characteristics of schizophrenia patients differing in learning proficiency on the Wisconsin Card Sorting Test. Schizophrenia Bulletin, 27, 687696.Google Scholar
Wiedl, K. H., Schöttke, H., Green, M. F., & Nuechterlein, K. H. (2004). Dynamic testing in schizophrenia. Does training change the construct validity of a test? Schizophrenia Bulletin, 30, 703711.CrossRefGoogle ScholarPubMed
Wilson, B., Cockburn, J., & Halligan, P. (1987). Development of a behavioural test of visuo-spatial neglect. Archives of Physical Medicine and Rehabilitation, 68, 98102.Google Scholar
Figure 0

Table 1 Characteristics of the participant groups

Figure 1

Fig. 1 Median dynamic Wisconsin Card Sorting Test scores for the patient subgroup divided into poor learners, strong learners, and high achievers.

Figure 2

Table 2 Descriptives, floor- and ceiling effects, and intercorrelations of dynamic Wisconsin Card Sorting Test scores of the dWCST group

Figure 3

Table 3 Spearman correlations between the dynamic Wisconsin Card Sorting Test pre-test, post-test score, and gain score, and Neuropsychological Tests of the dWCST group

Figure 4

Fig. 2 Median score for dynamic Wisconsin Card Sorting Test subgroup, repeated Wisconsin Card Sorting Test group, and the control group (dynamic Wisconsin Card Sorting Test).

Figure 5

Table 4 Between-group differences in neuropsychological test performance

Figure 6

Table 5 Between-group comparisons of dynamic and repeated Wisconsin Card Sorting Test scores