Introduction
Practice effects (PEs) are improvements in test performance due to prior test exposure (Beglinger, Tangphao-Daniels, et al., Reference Beglinger, Tangphao-Daniels, Kareken, Zhang, Mohs and Siemers2005; McCaffrey, Duff, & Westervelt, Reference McCaffrey, Duff and Westervelt2000). PEs are usually conceptualized as a combination of long-term memory (implicit and explicit) and learning of task characteristics, and are well-known confounds of serial assessment (Busch, Chelune, & Suchy, Reference Busch, Chelune and Suchy2006). However, a growing body of literature suggests that PE may have diagnostic utility as a unique cognitive construct. For example, in a meta-analysis of PE, Calamia, Markon, and Tranel (Reference Calamia, Markon and Tranel2012) found that PE magnitudes depend not only on logistical factors, such as inter-test interval or use of alternate forms (Beglinger, Gaydos, et al., Reference Beglinger, Gaydos, Tangphao-Daniels, Duff, Kareken, Crawford and Siemers2005; Benedict & Zgaljardic, Reference Benedict and Zgaljardic1998), but also vary by age (Dikmen, Heaton, Grant, & Temkin, Reference Dikmen, Heaton, Grant and Temkin1999) and diagnosis (Basso, Bornstein, & Lang, Reference Basso, Bornstein and Lang1999; Wilson, Watson, Baddeley, Emslie, & Evans, Reference Wilson, Watson, Baddeley, Emslie and Evans2000).
Support for diagnostic and prognostic utility of PEs is evident in studies examining PEs among patients with mild cognitive impairment (MCI) and dementia (Duff, Reference Duff2012; Duff et al., Reference Duff, Beglinger, Schultz, Moser, McCaffrey, Haase and Paulsen2007; Duff, Callister, Dennett, & Tometich, Reference Duff, Callister, Dennett and Tometich2012; Duff, Chelune, & Dennett, Reference Duff, Chelune and Dennett2012; Machulda et al., Reference Machulda, Pankratz, Christianson, Ivnik, Mielke, Roberts and Petersen2013). In this research, findings have been somewhat mixed. Some studies show that individuals with dementia and MCI have smaller PEs than healthy peers on measures of category fluency (Cooper et al., Reference Cooper, Epker, Lacritz, Weiner, Rosenberg, Honig and Cullum2001; Cooper, Lacritz, Weiner, Rosenberg, & Cullum, Reference Cooper, Lacritz, Weiner, Rosenberg and Cullum2004), episodic memory (Duff, Chelune, et al., Reference Duff, Chelune and Dennett2012; Schrijnemaekers, de Jager, Hogervorst, & Budge, Reference Schrijnemaekers, de Jager, Hogervorst and Budge2006), and cognitive status (Helkala et al., Reference Helkala, Kivipelto, Hallikainen, Alhainen, Heinonen, Tuomilehto and Nissinen2002), presumably due to memory impairments (Jonker, Geerlings, & Schmand, Reference Jonker, Geerlings and Schmand2000; Mitchell, Reference Mitchell2008). By contrast, others have observed greater PEs in MCI on measures of verbal and visual explicit memory (Duff et al., Reference Duff, Beglinger, Van Der Heiden, Moser, Arndt, Schultz and Paulsen2008) and motor control (Yan & Dick, Reference Yan and Dick2006). Such apparently paradoxical findings call into question the prevailing conceptualization of PE as reflecting memory and learning, and suggest that further investigation into the nature of PEs is warranted.
Several explanations have been offered for larger PEs among individuals with MCI, including floor/ceiling effects across patient groups, differential declines in declarative versus procedural learning, or heterogeneity in cognitive status within groups (Duff et al., Reference Duff, Beglinger, Van Der Heiden, Moser, Arndt, Schultz and Paulsen2008). Alternatively, PEs may reflect cognitive phenomena beyond memory, such as a rebound from initial transient decrements in performance caused by poor adaptation to novel task characteristics. That is, when faced with novel tasks, individuals with MCI may initially be overwhelmed by unfamiliar task characteristics, therefore, performing below their actual cognitive potential. Once familiar with the task, such individuals exhibit a rebound from this initial performance decrement. This rebound has been termed the “novelty effect” (Suchy, Kraybill, & Franchow, Reference Suchy, Kraybill and Franchow2011). When this rebound occurs during a second administration of the same task, it clearly contributes to PE. Because the rebound can only be as large as the initial decrement (i.e., the larger the decrement, the larger the rebound), individuals with MCI (who become more overwhelmed by novel tasks) exhibit larger rebound and, therefore, larger PEs.
Although both learning/memory and novelty effect are associated with performance improvements with repeated task exposure, novelty effect differs from memory in that it does not reflect acquired knowledge or skills relative to an initial baseline. Rather, novelty-related improvements reflect a recovery to baseline from an initial suppression of performance. Because novelty effect can be observed even on tasks that have been previously learned but are presented in novel contexts (Euler, Niermeyer, & Suchy, Reference Euler, Niermeyer and Suchy2015; Larson & Suchy, Reference Larson and Suchy2014; Ouellet, Beauchamp, Owen, & Doyon, Reference Ouellet, Beauchamp, Owen and Doyon2004), it appears to represent a construct that is distinct from memory. Whereas learning-related improvements reflect better ability to acquire and retain new knowledge or skill, improvements due to novelty effect reflect poorer rapid adaptation to novel task demands (e.g., manipulating novel materials or maintaining instructions in working memory).
Recently, we proposed a theoretical model of PE (Figure 1) to explain paradoxical PEs in MCI (Suchy et al., Reference Suchy, Kraybill and Franchow2011). We conceptualized PEs as consisting of at least two components: (1) memory (both implicit and explicit) and (2) novelty effect. The model posits that memory and novelty effect contribute differentially to PE at different points along the declining trajectory: Whereas the contribution of memory to PE generally declines when pathological cognitive change becomes apparent, the contribution of novelty increases early in the declining trajectory (possibly before pathological detectable memory change), and only later decreases as pathological declines continue. These differential contributions of memory and novelty jointly lead to a curvilinear relationship between PE and pathological cognitive decline (see Figure 1). While we have demonstrated larger novelty effect among individuals at preclinical stages of cognitive decline relative to non-declining counterparts (Suchy et al., Reference Suchy, Kraybill and Franchow2011), the remaining time points on the decline continuum are purely theoretical and are yet to be tested empirically. Furthermore, past research has not examined the direct association between novelty effect and PE or the differential contribution of novelty effect and memory along the cognitive decline continuum.
Fig. 1 Theoretical model of components of practice effect across a spectrum of cognitive decline. Adapted from Suchy et al. (Reference Suchy, Kraybill and Franchow2011); copyright 2011 by Cambridge University Press. This figure shows how practice effect (PE) can be conceptualized as the net sum of novelty effect and memory ability. According to the model, the initial increase in novelty effect is of sufficient magnitude to result in an increase in the net PE early on in the neurodegenerative process (i.e., when memory is still relatively preserved). Once a clinically significant level of cognitive decline is reached (i.e., when memory and learning begin to exhibit notable decrement), PE begins to decline, being comprised primarily of novelty effect with little contribution from memory. Finally, as cognitive impairments become more severe, novelty effect also declines because individuals are no longer able to rebound from the deleterious impact of novelty; thus, with minimal to no contributions from memory or novelty effect, PEs also become minimal or even non-existent.
The goal of this study was to examine PEs and their proposed components (i.e., memory and novelty effect) in older adults across a spectrum of cognitive decline with three primary aims. First (Aim 1), we tested the hypothesis that novelty effect and long-term explicit memory uniquely contribute to PE as predicted by our model. Second (Aim 2), we tested the hypothesized quadratic relationship between PE and cognitive decline, which predicts greater PE in the context of mild cognitive dysfunction relative to intact cognition or moderate-to-severe impairment. Third (Aim 3), we tested the hypothesis that memory and/or novelty effect would mediate the relationship between PE and cognitive decline.
Method
Participants and Recruitment
Participants included 75 adults ages 60 to 89 representing a continuum from healthy to severely impaired cognition. To ensure a range of cognitive functioning, participants were recruited from both the community (i.e., senior centers (n=2), assisted living facilities (n=1), and health fairs (n=43)] and a clinic at the University of Utah’s Center for Alzheimer’s Care, Imaging, and Research (n=29). Participants recruited from the community were an average of 3.3 years younger than those recruited from the clinic (p=.048), but did not differ on other demographic variables (see Table 1). Because our model hypothesizes that contributions of memory and novelty effect to PE change across the early stages of pathological cognitive decline, individuals exhibiting moderate-to-severe impairment on initial screening were excluded. Additional exclusion criteria were non–right-handednessFootnote 1 , severe depressive symptomsFootnote 2 , history of neurological disorder (e.g., stroke, seizures, moderate-to-severe brain injury), and serious psychiatric illness (e.g., psychosis, untreated depression). Of the 98 individuals screened, 82 met screening criteria and 7 withdrew before being scheduled for participation. Of the 75 participants who completed study procedures, two participants were excluded for severe depressive symptoms. Due to an administration error, one of the primary measures [Wechsler Adult Intelligence Scale, 4 th Edition (WAIS-IV) Coding] was not administered to the first seven study participants; therefore, these participants were removed from analyses. This left a final sample of 66 participants. Demographic characteristics of excluded versus included participants did not differ (ps=.34–.87). Four of the 66 included participants had scores that were potential outliers on a primary measureFootnote 3 . The results followed a similar pattern whether those participants were excluded or included in analyses; however, some of the results were reduced to trends, possibly due to low power (see the Results section for details). These cases were retained in the analyses to improve power.
Table 1 Descriptive statistics for sample
Note. TICS=Telephone Interview of Cognitive Status; GDS=Geriatric Depression Scale; DRS-2=Age and education adjusted scaled scores for the Mattis Dementia Rating Scale, 2nd edition; PECoding=practice effect calculated as difference between time 2 and time 1 raw scores on WAIS-IV Coding; PESearch=practice effect calculated as difference between time 2 and time 1 raw scores on WAIS-IV Symbol Search; Memory=Rey Auditory Verbal Learning Test delayed recall; Novelty Effect=difference in motor planning times between first and second blocks of a motor learning task.
Procedures
The study was approved by the University of Utah Institutional Review Board. Participants were pre-screened for inclusion/exclusion criteria via brief telephone interview regarding demographics, self-reported handedness, and medical history. Written informed consent was obtained from participants (and legally authorized representatives, if applicable) before participation. Participants completed an individually administered 2-hr battery of cognitive tasks, were fully debriefed upon completion, and were compensated $10 per hr.
Measures
Eligibility screening
The Telephone Interview of Cognitive Status (TICS; Brandt & Folstein, Reference Brandt and Folstein2003) was used to screen for cognitive status before enrollment. The TICS includes items similar to the Mini Mental State Exam (MMSE) and correlates highly (r=.94) with the MMSE. It also has excellent sensitivity (94%) and specificity (100%) for distinguishing demented from non-demented participants (Brandt, Spencer, & Folstein, Reference Brandt, Spencer and Folstein1988). Following Brandt and Folstein (2003) interpretive ranges, a cutoff score of 21 or above was selected with the goal of excluding individuals with moderate-to-severe impairmentFootnote 4 . Participants were screened for depression using the Geriatric Depression Scale (GDS; Yesavage, Reference Yesavage1982), which has good validity and reliability among community-dwelling older adults (Dunn & Sacco, Reference Dunn and Sacco1989; Yesavage, Reference Yesavage1982) and adults with mild to moderate dementia (Feher, Larrabee, & Crook, Reference Feher, Larrabee and Crook1992). We used a cutoff score of 19 or above (indicating severe depressive symptoms).
Cognitive decline
Abnormal cognitive decline was operationalized as deviation from demographically expected performance on the Mattis Dementia Rating Scale, 2 nd Edition (DRS-2; Mattis, 1988); thus, age and education adjusted scaled scores were used in all analyses. DRS-2 is a screening measure used to assess general cognitive decline and includes items assessing attention, initiation, abstraction, visual-constructional abilities, and memory. As explained in the DRS-2 manual, scaled scores of 11 and above represent “average” (i.e., normatively expected) or higher cognitive functioning while scaled scores of 10 and below represent progressively greater deviation from normative expectations (Mattis, 1988).
Practice effects
PEs were measured using repeated administration of the Symbol Search and Coding subtests of the WAIS-IV (Wechsler, Reference Wechsler2008), which are paper and pencil processing speed tasks with scores reflecting the number of items correctly completed within 2 min. These tests were selected because (1) they were not designed to assess memory or novelty effect (Wechsler, Reference Wechsler2008), and, therefore, would not be expected to confound contributions of memory and novelty effect to PE; (2) they are known to exhibit sizeable practice effects in normative samples as compared to, for example, measures of crystallized intelligence (Estevis, Basso, & Combs, Reference Estevis, Basso and Combs2012); and (3) they were presumed to assess the same construct regardless of repeated administrations, which is not the case for all cognitive measures. Following recent methods in PE research (Darby, Maruff, Collie, & McStephen, Reference Darby, Maruff, Collie and McStephen2002; Duff, Chelune, et al., Reference Duff, Chelune and Dennett2012), the same form of each measure was repeated within-session at 30-min intervals. Participants completed other measures during this interval.
Test–retest reliabilities within our sample were .840 for Symbol Search and .894 for Coding. PEs for each subtest were calculated as the change in raw scores between the first and second administrations (the second score minus the first score). These subtests were originally intended to be combined into a PE composite score to optimize reliability. However, the two PE variables were not correlated (see Table 2); therefore, they were examined separately in all analyses.
Table 2 Bivariate Pearson product correlations among dependent and independent variables (n=66)
*p<.05. **p<.01. †Lower values reflect better performances.
Note. DRS-2=Age and education adjusted scaled scores for the Mattis Dementia Rating Scale, 2nd edition; Memory=Rey Auditory Verbal Learning Test delayed recall; Novelty effect=difference in motor planning times between first and second blocks of a motor learning task; PECoding=practice effect for WAIS-IV Coding; PESearch=practice effect for WAIS-IV Symbol Search; GDS=Geriatric Depression Scale.
Memory
The Rey Auditory Verbal Learning Test (RAVLT; Schmidt, Reference Schmidt1996) is a 15-item list learning and memory task that includes five learning trials and a delayed (20–30 min) recall trial. The RAVLT has good test–retest reliability and validity (Schmidt, Reference Schmidt1996). Memory was operationalized as total number of items recalled on the delay trial, to be comparable to the delay used for assessment of PE.
Novelty effect
Following our prior work (Euler et al., Reference Euler, Niermeyer and Suchy2015; Suchy, Euler, & Eastvold, Reference Suchy, Euler and Eastvold2014; Suchy et al., Reference Suchy, Kraybill and Franchow2011), novelty effect was measured using the Push-Turn-Taptap (PTT) task (Suchy & Kraybill, Reference Suchy and Kraybill2007), an electronically administered sequence learning task from the Behavioral Dyscontrol Scale, Electronic Version (BDS-EV; Suchy, Derbidge, & Cope, Reference Suchy, Derbidge and Cope2005). Participants perform sequences of three hand movements across four blocks, using a response console (Figure 2). These sequences progressively increase in complexity across these four blocks from two movements (Block 1) to five movements (Block 4). Motor planning latencies (i.e., time elapsed between completion of one sequence and initiation of the next correct sequence) on the first block are typically affected by novelty, as they are longer than those on the second block. Novelty effect is operationalized as the difference in these latencies between the second and first blocks. In this sample, the reliability of motor planning latencies was .858.
Fig. 2 Response console for the Push-Turn-Taptap (PTT) task. Using this response console, participants perform sequences (or permutations) of three specified hand movements. These are (a) pushing the joystick forward, (b) turning the joystick clockwise, and (c) double-tapping on the large white dome button. The task consists of four blocks, with each block requiring that participants learn a new and progressively longer sequence (2 movements through 5 movements). Each block begins with 3 learning trials presented on a computer screen. After the learning trials, the computer screen goes blank and participants continue to perform the sequence from memory until 5 correct consecutive sequences have been executed. A block is terminated if the criterion of 5 correct trials is not accomplished after 10 trials. Mistakes are followed by an audible tone, along with a presentation of the correct sequence on the screen. Although the task consists of four blocks, only the first two blocks are used for the calculation of the novelty effect. Also, although the task automatically tracks speed and accuracy of all movements across all four blocks (allowing assessment of not only motor planning, but also motor sequence learning, motor control, and motor speed), only motor planning has been shown to exhibit the novelty effect in past research. Thus, only the motor planning variable was used in this study.
Statistical Analyses
Data were analyzed in SPSS using ordinary least squares regression. Independent contributions of novelty effect and memory to PE (Aim 1) were examined using hierarchical multiple regressions with PEs on WAIS-IV Symbol Search (PESearch) and Coding (PECoding) as criterion variables. RAVLT delayed recall and novelty effect were used as predictors at Steps 1 and 2, respectively, and subsequently reversed (i.e., Steps 2 and 1, respectively) to examine unique contributions to PE. To test the hypothesis that the relationship between abnormal cognitive decline and PE is curvilinear (Aim 2), we conducted multiple regressions using PESearch and PECoding as the criterion variables. Predictors included linear and quadratic terms for DRS-2 scaled scoresFootnote 5 .
To determine whether the relationships between PE and cognitive decline were mediated by learning or novelty effect (Aim 3), we used the MEDCURVE procedure for SPSS (Hayes & Preacher, Reference Hayes and Preacher2010) to estimate the total, direct, and indirect effects Footnote 6 of DRS-2 scores on PEs. The MEDCURVE procedure was designed for path models in which one or more paths is nonlinear and for nonlinear models provides estimates of indirect effects at specific values of an independent variable, called instantaneous indirect effects. To test for significance of indirect effects, the MEDCURVE procedure generates bias-corrected bootstrap-confidence intervals (CIs) for the indirect effects; CIs that do not include zero indicate significant effects. As recommended by Preacher and Hayes (2004), we used 5000 bootstrap samples to create 95% confidence intervals for estimates of indirect effects. Following recommendations by Hayes (Reference Hayes2009), mediation was interpreted when both the total effect (i.e., the path from a focal predictor to the criterion variable) and the indirect effect (i.e., the path from a focal predictor to the mediator to the criterion variable) reached significance. Additional detail about mediation analysis is presented in Appendix B.
Results
Preliminary Analyses
Table 2 shows Pearson product correlations between independent and dependent variables, demographics, and depression symptoms. Pairwise scatterplots for the primary variables of interest are available in Supplementary Materials. PESearch was positively correlated with novelty effect (r=.489; p<.001), but no other variables. In contrast, PECoding was positively correlated with RAVLT delayed recall (r=.298; p=.015) and DRS-2 (r=.354; p=.004), which were also positively correlated with each other (r=.674; p<.001). Additionally, age was negatively correlated with PECoding (r=−.270; p=.028) and RAVLT delayed recall (r=−.447; p<.001). As mentioned earlier, PECoding and PESearch were not correlated and thus were examined separately in primary analyses.
Aim 1: Contributions of Learning and Novelty Effect to Practice Effect
As seen in Table 3, RAVLT delayed recall accounted for unique variance in PECoding, whereas novelty effect accounted for unique variance in PESearch. In sum, consistent with our hypotheses, these results show that memory and novelty effects have unique effects on PE, although, unexpectedly, each contributed to PE on a different measure. As a supplement, we repeated these analyses including both age and education, which are typically considered as covariates of cognitive performance in clinical neuropsychology. Results (see Table 3) followed the same pattern as our principal analyses for PESearch. However, in the analysis of PECoding, neither novelty effect nor delayed recall were significant predictors after including age and education (age and education themselves did not predict PECoding either). This is likely due to high intercorrelations among age, delayed recall, and PECoding (see Table 2) and overlapping variance between age (semipartial r=−.160) and delayed recall (semipartial r=.197) in predicting PECoding.
Table 3 Novelty effect and memory as predictors of PE on WAIS-IV Coding and Symbol Search tests
Δ=change; df=degrees of freedom; PECoding=practice effect for WAIS-IV Coding; PESearch=practice effect for WAIS-IV Symbol Search; Memory=Rey Auditory Verbal Learning Test delayed recall; Novelty effect=difference in motor planning times between first and second blocks of a motor learning task.
Aim 2: Relationship between Cognitive Decline Status and Practice Effect
Results indicated a positive linear effect of DRS-2 scores on PECoding (linear term: b=.640; beta=.354; t=3.028; p=.004) with DRS-2 scores accounting for 12.5% of variance in PECoding. In contrast, DRS-2 scores showed a quadratic relationship with PESearch (quadratic term: b=−.119; beta=−.347; t=−2.532; p=.014)7, such that larger PESearch was associated with intermediate DRS-2 scores (i.e., mild impairment, approximate DRS-2 scaled score=8), whereas smaller PESearch was associated both with highest and lowest DRS-2 scores (see Figure 3B). DRS-2 scores accounted for 9.7% of variance in PESearch. Results followed a similar pattern with age and education included as covariates, indicating that these results cannot be explained by demographic factorsFootnote .
Fig. 3 Scatterplot of practice effects on WAIS-IV Coding (Panel A) and Symbol Search (Panel B) across cognitive decline.
In sum, consistent with our model, PESearch was a quadratic function of cognitive decline. However, rather than peaking at a preclinical level of decline, PESearch peaked at an approximate DRS-2 scaled score of 8, which is on the cusp of clinical impairment per DRS-2 normative standards (Mattis, 1988). In contrast, the relationship between DRS-2 scores and PECoding is consistent with the expected linear decrease in PE with cognitive decline.
Aim 3: Mediation Analyses
Because RAVLT delayed recall and novelty effect were differentially related to the two PE variables in Aim 1 analyses, separate mediation analyses were examined for each PE variable. The models in Table 4 were used to estimate total, direct and indirect effects of (1) DRS-2 scores on PECoding through delayed recall (Table 4; Figure 4A) and (2) DRS-2 scores on PESearch through novelty effect (Table 4; Figure 4B).
Fig. 4 Path coefficients for mediation models for practice effect on WAIS-IV Coding (A) and Symbol Search (B) tests (n=66). Note: Dotted lines indicate the effect of cognitive decline on PE when the mediator is excluded from the model. a, b, c, and c’ are unstandardized regression coefficients. The indirect effect of cognitive decline through the proposed mediator is calculated as the product of a and b for linear mediation models and as the product of the partial derivative of the models for paths a and b for nonlinear models. † p<.10, *p<.05, **p<.01, ***p<.001
Table 4 Regression results for models used in mediation analyses of cognitive decline on (A) WAIS-IV Coding practice effect through memory and (B) WAIS-IV Symbol Search through novelty effect
df=degrees of freedom; PECoding=practice effect for WAIS-IV Coding; PESearch=practice effect for WAIS-IV Symbol Search; Memory=Rey Auditory Verbal Learning Test delayed recall; Novelty=difference in motor planning times between first and second blocks of a motor learning task; i=intercept; c=total effect of DRS-2 on PE; a=direct effect of DRS-2 on mediator; b=direct effect of mediator on PE independent of DRS-2; c’=direct effect of DRS-2 on PE independent of mediator.
Note. The indirect effect of DRS-2 on PE through the proposed mediator is quantified as the product of a and b. The total effect of DRS-2 on PE is the sum of the direct and indirect effects: c=ab+c’. Using this equation, the indirect effect can be calculated as the difference between the total and direct effects of DRS-2 on PE: ab=c--c’.
Tests of direct effects
As shown in Table 5, DRS-2 scores had a significant linear total effect on PECoding (Model 1; Figure 4A, path c) and a significant linear direct effect on delayed recall (Model 2; Figure 4A, path a), which decreased with declining DRS-2 scores. The direct effects of delayed recall (Table 5, Model 3; Figure 4A, path b) and DRS-2 scores (Table 5, Model 3; Figure 4A, path c’) on PECoding were not significant. As seen in Table 4 (Analysis A), adding delayed recall as a predictor of PECoding increased explained variance from 12.5% to 13.2%.
Table 5 Mediation analysis for cognitive decline on WAIS-IV Coding practice effect through memory (standard errors in parentheses)
PECoding=practice effect on WAIS-IV Coding; DRS-2=age and education adjusted scaled scores for the Mattis Dementia Rating Scale, 2nd edition; Memory=Rey Auditory Verbal Learning Test delayed recall; c=total effect of DRS-2 on PECoding; a=direct effect of DRS-2 on memory; b=direct effect of memory on PECoding independent of DRS-2; c’=direct effect of DRS-2 on PECoding independent of memory.
Note. All coefficients are unstandardized ordinary least squares regression coefficients. Model 1: PECoding=intercept+c(DRS-2)+error. Model 2: Memory=intercept + a(DRS-2) + error. Model 3: PECoding=constant + c’(DRS-2) + b(Memory) + error.
Consistent with Aim 2 results, DRS-2 scores had a significant quadratic total effect on PESearch (Table 6, Model 1; Figure 4B, path c 2 )Footnote 8 . The direct effects of DRS-2 scores on novelty (Table 6, Model 2; Figure 4B, paths a 1 and a 2 ) and PESearch (Table 6, Model 3; Figure 4B, path c’ 2 ) were not significant. Novelty effect had a positive linear direct effect on PESearch (Table 6, Model 3; Figure 4B, path b). Additionally, adding novelty effect to the model, increased explained variance in PESearch from 9.7% to 28.6% (see Table 4, Analysis B). Results for path models of both PECoding and PESearch followed a similar pattern when covariates were included. Together these results are generally consistent with the Suchy et al. (2011) theoretical model.
Table 6 Mediation analysis for cognitive decline on WAIS-IV Symbol Search practice effect through novelty effect (standard errors in parentheses)
PESearch=practice effect on WAIS-IV Symbol Search; DRS-2=age and education adjusted scaled scores for the Mattis Dementia Rating Scale, 2nd edition; Novelty=novelty effect calculated as the difference in motor planning times between first and second blocks of a motor learning tas; c=total effect of DRS-2 on PESearch; a=direct effect of DRS-2 on novelty; b=direct effect of novelty on PESearch independent of DRS-2; c’=direct effect of DRS-2 on PESearch independent of novelty.
Note. All coefficients are unstandardized ordinary least squares regression coefficients.Model 1: PESearch=constant + c 1 (DRS-2) + c 2 (DRS-2)2 + error.
Model 2: Novelty Effect=constant + a 1 (DRS-2) + a 2 (DRS-2)2 + error. Model 3: PESearch=constant + c’ 1 (DRS-2) + c’ 2 (DRS-2)2 + b(Novelty) + error.
Tests of indirect effects
The estimate of the indirect effect of DRS-2 scores on PECoding was not significant (indirect effect=.132; 95% CI=−.153 to .437), which may have been partly due to high correlation between DRS-2 scaled scores and delayed recall (r=.674; p<.001), resulting in minimal unique variance in PECoding explained by delayed recall (semipartial r=.080). Next we examined novelty effect as a mediator of the effect of DRS-2 scores on PESearch. As shown in Figure 5, novelty effect partially mediated the relationship between PESearch and DRS-2 with significant instantaneous indirect effects for DRS-2 scaled scores of 7 and below (impaired status; θ DRS-2=7 =.126; 95% CI=.007 to .349). Figure 5 displays instantaneous indirect effects for all DRS-2 scores. When covariates were included in the model, the general pattern of results was similar for both PECoding and PESearch. However, in the latter, mediation by novelty effect occurred only for DRS-2 scores of 6 and below (θ DRS-2=6 =.188; 95% CI=.007 to .569).
Fig. 5 Instantaneous indirect effects of cognitive status on WAIS-IV Symbol Search practice effect mediated by novelty effect. Dashed lines represent upper and lower confidence intervals for bootstrap-estimated instantaneous direct effects. Confidence intervals that do not include zero indicate a significant indirect effect. Significant mediation is depicted by solid bold line.
These results indicate (1) that novelty effect accounted for the effects of DRS-2 scores on PESearch at impaired levels of cognitive functioning, and (2) that these relationships are not due to demographic factors. However, novelty effect did not explain effects of DRS-2 scores on PESearch for cognitively intact participants. Taken together, the mediation analyses suggest that changes in PE with cognitive decline may be attributable to specific cognitive processes that may vary depending on the measures on which PEs are observed.
Discussion
The key findings of this study were that (1) PE is not a unitary construct and, depending on how it is assessed, it may be explained by memory, novelty effect, or both; (2) the relationship between PE and cognitive decline may be linear for some, and curvilinear for other, measures of PE; and (3) the relationship between cognitive decline and PE on WAIS-IV Symbol Search may be explained by novelty effect, particularly at impaired levels of cognitive functioning. While some aspects of these results were consistent with the original hypotheses and partially supported the theoretical model of PE, others were unexpected. Our results do support contributions of both memory and novelty effect to PE as proposed in the theoretical model. Additionally, our results are consistent with the notion that novelty and cognitive decline may have a nonlinear relationship, and that novelty effect may partially mediate nonlinear changes in PE as a function of cognitive decline. However, contrary to expectation, our two measures of PE were not correlated with each other, and, therefore, needed to be analyzed separately. These separate analyses revealed that the two PEs were uniquely related to memory and novelty effect, such that memory predicted PE on WAIS-IV Coding (PECoding) whereas novelty effect predicted PE on WAIS-IV Symbol Search (PESearch).
Although previous research has shown substantial variability in magnitudes of PE across different cognitive domains (Basso, Carona, Lowery, & Axelrod, Reference Basso, Carona, Lowery and Axelrod2002; Duff et al., Reference Duff, Beglinger, Moser, Paulsen, Schultz and Arndt2010, Reference Duff, Beglinger, Van Der Heiden, Moser, Arndt, Schultz and Paulsen2008), the fact that our two indices of PE were uncorrelated is nevertheless unexpected given that they were observed on measures of the same cognitive domain. One interpretation of this finding is that each measure draws upon different component processes beyond speed, and these processes are then differentially facilitated by practice. Indeed, memory for number-symbol pairs appears to facilitate performance on WAIS-IV Coding above and beyond speed (Joy, Fein, & Kaplan, Reference Joy, Fein and Kaplan2003; Joy, Fein, Kaplan, & Freedman, Reference Joy, Fein, Kaplan and Freedman2000; Joy, Kaplan, & Fein, Reference Joy, Kaplan and Fein2004). In contrast, memory processes would offer little support on Symbol Search retest, which may rely more on executive and visual processing (Sweet et al., Reference Sweet, Paskavitz, O’Connor, Browndyke, Wellen and Cohen2005).
Our findings of unique contributions of cognitive processes to PE and different patterns (i.e., linear vs. nonlinear) of PEs with cognitive decline help explain mixed results in the literature regarding PE and cognitive decline (Cooper et al., Reference Cooper, Lacritz, Weiner, Rosenberg and Cullum2004; Duff et al., Reference Duff, Beglinger, Van Der Heiden, Moser, Arndt, Schultz and Paulsen2008; Duff, Chelune, et al., Reference Duff, Chelune and Dennett2012; Yan & Dick, Reference Yan and Dick2006), and suggest that differences in PE between impaired and nonimpaired groups depend on the specific measure used. For example, cognitive impairment is likely associated with smaller PE on tests of memory (Schrijnemaekers et al., Reference Schrijnemaekers, de Jager, Hogervorst and Budge2006; but see Duff et al., Reference Duff, Beglinger, Van Der Heiden, Moser, Arndt, Schultz and Paulsen2008 for contradictory results), but larger PE on other measures, such as motor control tasks (e.g., Yan & Dick, Reference Yan and Dick2006). This notion is further supported by the results of our mediation analyses wherein novelty effect partially mediated the effect of cognitive decline on PESearch.
Theoretical Implications
Our finding that PE has diverse cognitive underpinnings is consistent with prior research showing residual PE on alternate forms despite changes in test content (Beglinger, Gaydos, et al., Reference Beglinger, Gaydos, Tangphao-Daniels, Duff, Kareken, Crawford and Siemers2005; Benedict, Reference Benedict2005; Benedict & Zgaljardic, Reference Benedict and Zgaljardic1998). These residual PEs could reflect implicit memory processes and other cognitive phenomena, including novelty effects. For example, after observing larger PEs on a visual memory task versus a list-learning task, Benedict and Zgaljardic (Reference Benedict and Zgaljardic1998) noted that in addition to differing in verbal versus nonverbal memory demands, the procedures of the nonverbal memory task were more novel to participants relative to the familiar list-learning procedures.
While the novelty effect appears to be distinguishable from explicit memory, both in this study and in our prior research (Suchy et al., Reference Suchy, Kraybill and Franchow2011), we currently have a poor conceptual understanding of novelty. It is possible that novelty effect merely reflects implicit/procedural learning, which is dissociable from explicit memory (Squire, Reference Squire1994) and may be relatively preserved in MCI and early Alzheimer’s disease (Akdemir, Cangöz, Örsel, & Selekler, Reference Akdemir, Cangöz, Örsel and Selekler2007; Gobel et al., Reference Gobel, Blomeke, Zadikoff, Simuni, Weintraub and Reber2013). Consequently, MCI patients are able to exhibit the rebound in performance that reflects novelty effect. Alternatively, novelty effect could reflect other cognitive processes, such as controlled attention or strategy selection. While these have not been examined directly, several lines of research (detailed below) offer insights into the possible correlates of novelty effects.
Novelty effect may reflect specific aspects of executive functioning, such as controlled attention, which are involved in set formation or shifting. For example, learning curves research has shown a ubiquitous exponential performance pattern marked by large improvements within the first few trials of a task (Heathcote, Brown, & Mewhort, Reference Heathcote, Brown and Mewhort2000; Newell & Rosenbloom, Reference Newell and Rosenbloom1981), often referred to as a fast-learning stage, which is akin to our definition of novelty effect. This initial learning stage is thought to relate to attention, response selection, and mapping of response to stimuli (Halsband & Lange, Reference Halsband and Lange2006). In addition, temporary performance decrements (i.e., slower responses and/or increased errors) are consistently observed in task-switching paradigms (e.g., Biederman, Reference Biederman1972; Rogers & Monsell, Reference Rogers and Monsell1995) or in response to the reorganization of previously rehearsed task items (Ouellet et al., Reference Ouellet, Beauchamp, Owen and Doyon2004). Novelty effects may also be related to fluid intelligence as lower fluid intelligence is associated with larger PE (Blalock & McCabe, Reference Blalock and McCabe2011). Lastly, we recently demonstrated that novel contexts lead not only to behavioral novelty effects, but also to degradation of the EEG-assessed motor readiness potential (Euler et al., Reference Euler, Niermeyer and Suchy2015), suggesting that the ability to overcome novelty may reflect the efficiency of neuronal synchronization in face of the distracting properties of novel contexts.
Clinical Implications
We recently proposed that novelty effect may represent an early preclinical marker of declining cognitive reserve, the cognitive “buffer” that protects against behavioral manifestations of neurodegenerative disease (Suchy et al., Reference Suchy, Kraybill and Franchow2011). Cognitive reserve may mask cognitive decline through greater activation or broader recruitment of brain regions to support performance of novel tasks (Eyler, Sherzai, Kaup, & Jeste, Reference Eyler, Sherzai, Kaup and Jeste2011; Lenzi et al., Reference Lenzi, Serra, Perri, Pantano, Lenzi, Paulesu and Macaluso2011). However, activation of broader neural networks may lead to subtle costs early on in task performance; these costs may take the form of delayed re-emergence of motor readiness potentials, which then results in longer latencies before response initiation (Euler et al., Reference Euler, Niermeyer and Suchy2015). While the present results provide support for the clinical utility of PE (Duff, Reference Duff2012; Duff et al., Reference Duff, Beglinger, Schultz, Moser, McCaffrey, Haase and Paulsen2007; Duff, Callister, et al., Reference Duff, Callister, Dennett and Tometich2012; Duff, Chelune, et al., Reference Duff, Chelune and Dennett2012; Machulda et al., Reference Machulda, Pankratz, Christianson, Ivnik, Mielke, Roberts and Petersen2013) and novelty effect (Suchy et al., Reference Suchy, Kraybill and Franchow2011), they also demonstrate needs for future research examining PE as a non-unitary construct and tailoring PE assessment to different clinical populations.
Variations in the relationship between PE and cognitive decline across cognitive measures could have implications for interpretation of serial assessments. For example, while large PEs may indicate intact or improved cognitive functioning on some measures (e.g., learning/memory measures), they may represent impairment or an incipient neurodegenerative disorder on measures with more novel task demands. Reliable change indices (RCI) have been developed to address practice-related variance in repeat test performance (Chelune & Franklin, Reference Chelune and Franklin2003; Duff, Reference Duff2012); however, RCIs are typically calculated using test–retest data from healthy samples, which may not accurately reflect retest variability in impaired populations.
Limitations
First, our ability to detect a mediating effect of memory on the relationship between cognitive decline and PE was limited by high correlations between measures of memory and cognitive decline, leading to overlapping variance in prediction of PECoding. This may relate to the fact that DRS-2 scores are heavily weighted on memory performance and thus correlate highly with memory tests (Smith, Ivnik, Malec, & Kokmen, Reference Smith, Ivnik, Malec and Kokmen1994). Future research should examine these relationships using other indices of cognitive decline, or populations whose cognitive decline is characterized by other changes. It remains to be seen whether different patterns of cognitive decline have differential impact on PE; however, preliminary support for this idea is evident in studies demonstrating that PEs vary across clinical diagnoses (Duff et al., Reference Duff, Beglinger, Schultz, Moser, McCaffrey, Haase and Paulsen2007).
Second, cognitive decline was not measured directly, but was estimated using age- and education-adjusted scaled scores. While such scores provide an estimate of deviation from premorbid expectation, low scaled scores may represent longstanding below-average functioning for some participants. Therefore, a direct assessment of cognitive change via longitudinal design is warranted.
Lastly, because a large portion of our sample was recruited from a memory disorders clinic, 26 participants (1 healthy, 25 with cognitive decline) had previously completed neuropsychological evaluations, which included the same or similar measures as those used in the current study. Thus, prior exposure could have led to artificially smaller PEs on the WAIS-IV Coding and Symbol Search tests. However, this effect could not explain the pattern of results, as all participants would have experienced equal exposure across both tests used for PE calculations.
Acknowledgments
There are no conflicts of interest to report. This study was financially supported by the University of Utah, Department of Neurology (G.J.C. & Y.S.) and Department of Psychology (S.R.T.)
Supplementary Material
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/S1355617715001332
APPENDIX A
Detailed Description of Measures
Telephone Interview of Cognitive Status (TICS; Brandt & Folstein, 2003). The TICS is a 10-min cognitive status screening measure with items assessing attention/concentration, orientation, single-trial list learning, serial subtraction, object naming, sentence repetition and simple reasoning. Scores range from 0 to 41 with four suggested interpretive ranges, including nonimpaired (scores 33 and above), ambiguous (26–32), mildly impaired (21–25), and moderately to severely impaired (20 or less).
Mattis Dementia Rating Scale, 2 nd Edition (DRS-2; Mattis, Reference Mattis1988). The DRS-2 is a brief paper and pencil screening measure for assessment of general cognitive decline. Test items assess domains of attention, initiation, abstraction, visual-constructional abilities, and verbal and nonverbal memory. The DRS-2 normative sample consists of 623 community-dwelling older adults from the Mayo’s Older Americans Normative Studies project (Lucas et al., 1998) who did not have any current medical, neurological, or psychiatric diagnoses that might impact cognitive functioning.
Wechsler Adult Intelligence Scale, 4 th Edition (WAIS-IV; Wechsler, 2008) Symbol Search and Coding Subtests. These pencil and paper tasks are measures of information processing speed and make up the Processing Speed Index of the WAIS-IV. Scores for both tests are based on the number of items completed within a 2-min time limit. For each item in the Symbol Search subtest participants were asked to identify one of two abstract symbols among a group of distractor symbols or to mark a box indicating that neither symbol is present. The Symbol Search Raw score is calculated as the number of correct items minus the number of incorrect items completed within the time limit (incomplete items are not scored). The Coding subtest is a symbol digit substitution test in which participants rapidly match an abstract symbol to numbers that are presented in a pseudorandom order. The Coding raw score is calculated as the total number of symbols correctly coded within the time limit.
References
Brandt, J., & Folstein, M. F. (2003). TICS, telephone interview for cognitive status: Professional manual. Odessa, FL: Psychological Assessment Resources.
Lucas, J. A., Ivnik, R. J., Smith, G. E., Bohac, D. L., Tangalos, E. G., Kokmen, E., ... Petersen, R. C. (1998). Normative data for the Mattis dementia rating scale. Journal of Clinical and Experimental Neuropsychology, 20(4), 536–547.
Mattis, S. (1988). Dementia rating scale: Professional manual. Odessa, FL: Psychological Assessment Resources.
Wechsler, D. (2008). Wechsler Adult Intelligence Scale-Fourth Edition. San Antonio, TX: Pearson.
APPENDIX B
Explanation of Mediation Analyses
Simple mediation analysis partitions the total effect of an independent variable X on a dependent variable Y into two separate components: the direct effect and the indirect effect. The direct effect of X represents the effect of X on Y that is independent of the proposed mediator, M. The indirect effect is the effect of X on Y that is accounted for by M. These effects are estimated using the following set of regression equations:
$$Y{\equals}i_{l} {\plus}cX$$
$$M{\equals}i_{2} {\plus}aX$$
$$Y{\equals}i_{3} {\plus}bM{\plus}c'X$$
where c is an estimate of the total effect of X on Y, a is an estimate of the direct effect of X on M, b is the direct effect of M on Y independent of X, and c’ is an estimate of the direct effect of X on Y independent of M. The indirect effect of X on Y through M is quantified as the product of a and b, which represents the rate at which Y changes as a function of both X and X’s effect on M. Thus the total effect of X on Y is the sum of the indirect and direct effects: c=ab+c’. Using this equation, one can also calculate the indirect effect as the difference between c and c’ (c--c’=ab). For a detailed explanation of these concepts see Hayes and Preacher (2010).
In contrast to linear mediation models where the indirect effect is constant for all values of X, in nonlinear models the indirect effect changes across values of X. For nonlinear models, the rate at which a change in X changes Y indirectly through changes in M is called an instantaneous indirect effect (denoted by θ x ) and represents the simple slope of the quadratic function at a particular value of X. To test for significance of the instantaneous indirect effect in nonlinear models, the MEDCURVE procedure enables computation of θ x and associated CIs for specified values of X. Instantaneous indirect effects are calculated the product of the partial derivative of the direct effect of X on M and the direct effect of M on Y using the following formula:
$$\theta {\equals}\left( {{{\partial M} \over {\partial X}}} \right)\left( {{{\partial Y} \over {\partial M}}} \right).$$
The mediation models examined in our study included a simple linear model as depicted in Equations 1–3 above and a nonlinear model in which the functions of Y and M with respect to X were quadratic:
$$Y{\equals}i_{1} {\plus}c_{1} X{\plus}c_{2} X^{2} $$
$$M{\equals}i_{2} {\plus}a_{1} X{\plus}a_{2} X^{2} $$
$$Y{\equals}i_{3} {\plus}bM{\plus}c'_{1} X{\plus}c'_{2} X^{2} .$$
Applying these models to Equation 4 yields the following formula for the instantaneous indirect effect, θ x :
$$\theta _{x} {\equals}(a_{1} {\plus}2a_{2} X)b$$
We used the MEDCURVE procedure for SPSS developed by Hayes & Preacher (2010) to estimate total, direct, and indirect effects for our hypothesized mediation models. This procedure is applicable to both linear and nonlinear models. The MEDCURVE procedure provides a test of significance for indirect effects by generating bias-corrected bootstrap confidence intervals (CIs) for the indirect effects. The bootstrapping procedure uses sampling with replacement to generate a large number of samples (with n equal to that of the original sample size) from the original data and computes CIs for the indirect effect. CIs that do not include zero indicate significant results. The bootstrapping method provides a more accurate test of significance of the indirect effect because it does not assume that the variables are normally distributed and it can be applied to small samples (Preacher & Hayes, 2004).
References
Hayes, A. F., & Preacher, K. J. (2010). Quantifying and testing indirect effects in simple mediation models when the constituent paths are nonlinear. Multivariate Behavioral Research, 45(4), 627–660. doi:10.1080/00273171.2010.498290
Preacher, K. J., & Hayes, A. F. (2004). SPSS and SAS procedures for estimating indirect effects in simple mediation models. Behavior Research Methods, Instruments & Computers, 36(4), 717–731. doi:10.3758/BF03206553
