Introduction
The broad aim of pediatric neuropsychological assessment is to provide a comprehensive evaluation of neurocognitive development and to establish profiles of strengths and weaknesses that quantify brain-behavior relationships. By definition, this suggests the inclusion of a broad number of domains of functioning in assessment protocols and their corresponding test batteries and measures. In some cases, multiple subtests measuring the same or similar functions are necessary to assess individuals’ performances. Although such assessments are designed to provide a broad overview of a child’s abilities, they also generate a large number of observations that ultimately need to be analyzed, interpreted, and integrated into a logical, evidence-based conclusion. Despite advances in the field and a certain commonality in the way neuropsychological assessments are performed, no universally accepted definition exists to identify neuropsychological impairment.
The absence of a clear definition may constitute a problem in clinical settings when lengthy, personalized test batteries are used. In such cases, conclusions about neuropsychological impairment may be based on the presence of limited and isolated low test scores. The lack of a quantitative definition also poses a challenge in research settings where scientists need to determine whether an individual is considered “impaired” or not, or when impairment needs to be defined in specific clinical populations (e.g., a dichotomized decision rule). This is especially true for pediatric brain injuries where there is substantial heterogeneity in the nature and degree of impairment (Beauchamp & Anderson, Reference Beauchamp and Anderson2013; Hung et al., Reference Hung, Carroll, Cancelliere, Cote, Rumney, Keightley and Cassidy2014; Yeates, Reference Yeates2010). Studies of neuropsychological outcome after pediatric brain injury typically report outcomes separately across a range of measures and domains without inferring whether individual profiles of performance reflect general neurocognitive impairment (Anderson et al., Reference Anderson, Spencer-Smith, Leventer, Coleman, Anderson, Williams and Jacobs2009; Conklin, Salorio, & Slomine, Reference Conklin, Salorio and Slomine2008; Levin et al., Reference Levin, Li, McCauley, Hanten, Wilde and Swank2013; Rieger et al., Reference Rieger, Lewandowski, Callahan, Spenceley, Truckenmiller, Gathje and Miller2013). Clinical versus control group comparisons provide evidence regarding the effect of brain injury on neurocognitive performance, but some research designs preclude such comparisons. Central to the purpose of the current study is the difficulty of defining neuropsychological impairment to identify a “case” (i.e., an individual with a particular clinical condition who demonstrates neuropsychological impairment) when no comparison group is present.
The existing literature suggests caution when interpreting multiple test scores to identify true cognitive impairment (Binder, Iverson, & Brooks, Reference Binder, Iverson and Brooks2009; Brooks & Iverson, Reference Brooks and Iverson2012; Crawford, Garthwaite, & Gault, Reference Crawford, Garthwaite and Gault2007; Hurks, Hendriksen, Dek, & Kooij, Reference Hurks, Hendriksen, Dek and Kooij2013). When interpreting a single test score in isolation, Gaussian distributions (bell curve) apply and clinicians can be confident that the rate of impairment is defined by the individual test score cutoff (i.e., 10% impairment rate when cutoff is at the 10th percentile). However, as mentioned above, neuropsychological assessments involve numerous observations or test scores that are not interpreted in isolation. Brooks and Iverson (Reference Brooks and Iverson2012) summarize several principles for consideration when interpreting multiple test scores to minimize the chances of a false positive, including: (1) test-score scatter or variability is common (i.e., it is uncommon to have all scores fall at the same level); (2) having some low scores is common (i.e., all cognitive batteries will have a “normal” number of low scores); (3) the number of low scores is related to the cutoff score used for interpretation (i.e., using a more liberal cutoff score will result in more low scores and a higher false positive rate); (4) the number of low scores depends on the number of tests administered (i.e., more tests administered and interpreted will result in more low scores and a higher false positive rate); and (5) the number of low scores varies by characteristics of the examinees (i.e., persons with lower intelligence or fewer years of education are likely to have more low scores than those with higher intelligence or more years of education).
Without adequate consideration of these principles, clinicians and researchers are at risk of higher-than-desired rates of suggested cognitive impairment when interpreting multiple neuropsychological scores in children and adolescents (Brooks, Reference Brooks2010; Brooks, Iverson, Feldman, & Holdnack, Reference Brooks, Iverson, Feldman and Holdnack2009; Brooks et al., Reference Brooks, Iverson, Koushik, Mazur-Mosiewicz, Horton and Reynolds2013; Brooks, Iverson, Sherman, & Holdnack, Reference Brooks, Iverson, Sherman and Holdnack2009; Brooks, Iverson, Sherman, & Roberge, Reference Brooks, Iverson, Sherman and Roberge2010; Brooks, Sherman, & Iverson, Reference Brooks, Sherman and Iverson2010; Crawford et al., Reference Crawford, Garthwaite and Gault2007; Hurks et al., Reference Hurks, Hendriksen, Dek and Kooij2013). Although these principles add another layer of interpretation with multiple test scores, understanding and considering multivariate principles may provide an opportunity for developing empirically derived criteria for identifying abnormal neuropsychological profiles (see Brooks, Iverson, Feldman, & Holdnack, Reference Brooks, Iverson, Feldman and Holdnack2009, for an example of the development and application of criteria for identifying cognitive impairment).
To date, there is no universally accepted definition of neuropsychological impairment. Elaborating such a definition would assist in identifying individual cases of impairment. The objective of this study was to empirically derive a Neuropsychological Impairment (NPI) rule to provide a definition that includes a range of functional and cognitive domains and which may be applied to children between the ages of 6 and 18 years, regardless of etiology or nature of neurological, developmental or psychiatric disorders. The NPI rule was derived on the basis of existing normative pediatric neuropsychological data, collected in the National Institutes of Health Pediatric Database project (NIHPD, Evans & Brain Development Cooperative Group, Reference Evans2006). Associations with parental income and education were explored to verify the hypothesis that impairment varies as a function of examinee characteristics.
Methods
Reference Group
A normative reference group was created using participants selected from the NIHPD pertaining to the “first objective” of the MRI Study of Normal Brain Development (Evans & Brain Development Cooperative Group, Reference Evans2006). This sample includes healthy children between the ages of 4 years, 6 months and 18 years, 3 months (at time of enrolment) recruited across multiple sites using community based sampling to reflect the demographic composition of the U.S. population with respect to sex, socioeconomic status (SES), and race/ethnicity and, including both males and females and right- and left-handed individuals. Participants were carefully screened for potential exclusionary criteria, which included any history or evidence of medical conditions that impact healthy brain development (see Evans & Brain Development Cooperative Group, Reference Evans2006, for details). In total, 433 children were enrolled across six Pediatric Study Centers. In the current study, only children between 6 and 18 years were included and the neuropsychological results from the initial assessment were used, resulting in a sample of 370 individuals. See Table 1 for the participant demographic characteristics. This research was completed in accordance with the Helsinki Declaration and institutional research standards for human research.
Table 1. Characteristics of the NIHPD sample
Note: Group 1=6–8 years, Group 2=9–12 years, Group 3=13–18 years.
a The average IQ level for the NIHPD sample (M=110.7, SD=12.5, 95% CI [109.4, 111.9]) was significantly higher than the normative mean of 100 (SD=10) (p<.001 by a one-sample t-test).
Outcome Measures
In the NIHPD study, outcome measures were chosen to allow correlations with neuroimaging markers and selection was guided by three requirements: (1) Description of cognitive and sensory motor abilities; (2) Description of academic skills; (3) Quantification of memory and executive functions (Complete protocol details can be found at http://pediatricmri.nih.gov/nihpd/info/protocols.html, see also Waber et al., Reference Waber, De Moor, Forbes, Almli, Botteron, Leonard and Group2007). To meet the first requirement, standardized neuropsychological tests available for use by others were chosen and included the following: Wechsler Abbreviated Scale of Intelligence (WASI, Wechsler, Reference Wechsler1999), Purdue Pegboard (Tiffin, Reference Tiffin1968; Tiffin & Asher, Reference Tiffin and Asher1948), and NEPSY verbal fluency (Korkman, Kirk, & Kemp, Reference Korkman, Kirk and Kemp1998). The second requirement was met by including three sub-tests from the Woodcock-Johnson-III Achievement Battery and the Pre-School Language Scale (Mather & Woodcock, Reference Mather and Woodcock2001; Zimmerman, Steiner, & Pond, Reference Zimmerman, Steiner and Pond2002). For the third requirement, verbal learning and memory was assessed using the California Verbal Learning Test (Delis, Kramer, Kaplan, & Ober, Reference Delis, Kramer, Kaplan and Ober1994), working memory was assessed using the Spatial Span and Spatial Working Memory subtests of the Cambridge Neuropsychological Test Automated Battery (CANTAB, Sahakian et al., Reference Sahakian, Morris, Evenden, Heald, Levy, Philpot and Robbins1988; Sahakian & Owen, Reference Sahakian and Owen1992), and executive functions were quantified using the CANTAB Intra/Extra-dimensional (ID/ED) set-shifting subtest. No basic measure of attentional skills was included in the NIHPD study. Instead, the presence of attentional difficulties was indirectly measured using the Attention Problems subscale of the Child Behavior Checklist (Achenbach, Reference Achenbach1991; Evans & Brain Development Cooperative Group, Reference Evans2006).
The set of subtests used for derivation of the NPI rule was drawn from the original NIHPD test battery and established by a national group of Canadian experts with the following constraints: (1) Ability to evaluate outcome on a range of neuropsychological domains; (2) Length of assessment time limited to approximately 2 hr to ensure feasibility in both clinical and research settings; (3) Target functions that are known or suspected to be altered in a range of neurodevelopmental and acquired conditions. Academic measures were not included in the initial derivation dataset because these skills are often secondarily affected by primary deficits in core cognitive functions. We chose not to include scores derived from parental questionnaires, such as the Child Behavior Checklist, because they are more subject to bias and are a reflection of parental perception rather than a direct observation of performance. We used the CANTAB Spatial Span subtest to represent the attentional domain in the current analyses. Although this subtest is considered to measure working memory, it overlaps with selective attention skills (Gazzaley & Nobre, Reference Gazzaley and Nobre2012). The final list of subtests for the rule derivation covered six domains of neuropsychological functioning (IQ, Attention/Working memory, Memory, Executive functions, Processing speed, Motor skills) using eight measures (see Table 2).
Table 2. List of the NIHPD domains and subtests used for rule derivation and internal validation and comparable subtests used from Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) for external clinical validation
a The CANTAB spatial span was used in the rule derivation as an indicator of attention.
b In the NIHPD sample verbal fluency was only administered to children 7 years and older, therefore the case definition for Group 1 (6–8 years) was based on 6 domains with 7 subtests, rather than 8 subtests in the older age groups.
c For external clinical validation, data was used with permission from : Yeates, K. O., Taylor, H. G., Wade, S., Drotar, D., Stancin, T., & Minich, N. (2002). A prospective study of short- and long-term neuropsychological outcomes after traumatic brain injury in children. Neuropsychology, 16(4), 514–523. Please see original manuscript for test descriptions and references.
CANTAB=Cambridge Neuropsychological Test Automated Battery; CELF=Clinical Evaluation of Language Fundamentals; CNT=Contingency Naming Test; CPT=Conners Performance Test; CVLT=California Verbal Learning Test; COWA=Controlled Oral Word Association Test; NEPSY=Developmental Neuropsychological Assessment; WASI=Wechsler Abbreviated Scale of Intelligence; WISC=Wechsler Intelligence Scale for Children-III; WJ=Woodcock-Johnson Tests of Cognitive Abilities.
Analyses
All statistical analyses were performed using SPSS version 21 (IBM Corp. Released 2012. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.).
The NIHPD study included both experimental and normed tests, resulting in a mix of standard (M=100; SD=10), scaled (M=10; SD=3), and raw scores for the key variables. In light of this, individuals were separated into three age groups to ensure greater comparability of performance. The three groups (early school period, Group 1: 6–8 years; middle school, Group 2: 9–12 years; adolescence, Group 3: 13–18 years) were formed using a developmental rationale reflecting cerebral growth spurts (Casey, Jones, & Hare, Reference Casey, Jones and Hare2008; Huttenlocher, Reference Huttenlocher1979; Huttenlocher & Dabholkar, Reference Huttenlocher and Dabholkar1997; Kolb, Pellis, & Robinson, Reference Kolb, Pellis and Robinson2004) and trajectories of memory and executive function maturation (Gathercole, Reference Gathercole1998). Similar age groups have also been used in clinical research on brain injury (Anderson et al., Reference Anderson, Spencer-Smith, Leventer, Coleman, Anderson, Williams and Jacobs2009; Sady, Vaughan, & Gioia, Reference Sady, Vaughan and Gioia2014; Zemek, Osmond, Barrowman, & Pediatric Emergency Research Canada Concussion Team, Reference Zemek, Osmond and Barrowman2013).
Standard and scaled scores were used for subtests when available, otherwise raw scores were used. The distributional properties of scores were assessed graphically using qq-plots, histograms, and boxplots. The distributions of the raw scores were found to deviate from normality (Shapiro-Wilk test, CANTAB ID/ED: W=.93; p<.001; CANTAB spatial span: W=.93; p<.001; CVLT trials 1–5: W=.96; p<.001; Pegboard: W=.97; p<.001), therefore, cutoffs were based on percentiles rather than means and standard deviations (SDs) for these scores. Data distributions for all subtests were checked for outliers with an interest primarily in underperforming individuals. No individuals were excluded on this basis. Frequency distributions were generated and the number of individuals performing at three different cutoff levels (1, 1.5, and 2 SDs below the mean, or the equivalent percentiles based on a normal distribution, namely the 15.9th, 6.7th, and 2.3rd percentiles) were identified by age group for each subtest. Our goal was to classify roughly 95% of the population as “typically functioning,” as this arbitrary definition offers a relatively strict specification of “caseness.”
Following rule derivation, the performance of the Neuropsychological Impairment Rule (NPI rule) was studied in three ways. First, the proportion of individuals impaired as a function of measure and domain was reported and these proportions were compared using McNemar’s tests with Holm-Bonferroni adjustments. Second, to explore putative links between socioeconomic factors and neuropsychological performance, the distribution of number of indicators of impairment was generated according to level of parental education and income and associations between these variables were assessed using the linear-by-linear χ2 test. The size of the effect of these comparisons was assessed via spearman rank correlation. Third, validation of the rule was conducted as follows: (1) Bootstrap validation: Bootstrap analysis was performed with 1000 samples to demonstrate the expected variability in classification of impairment; (2) Internal validation: Additional subtests were added to the battery from the NIHPD dataset to test the performance of the rule when the number of functional domains and variables increases. A dataset containing 10 variables covering eight domains of functioning was created by removing the WASI full scale IQ variable and replacing it by two of its components pertaining to language (Vocabulary) and visuo-constructive skills (Block Design), as well as adding in a test of academic functioning (Woodcock-Johnson Letter-Word Identification, Mather & Woodcock, Reference Mather and Woodcock2001) (see Table 2 for details of all datasets and versions used in the derivation and validation of the NPI rule). An extended battery was then created by increasing the number of variables, and therefore the number of correlations, but keeping the number of domains of functioning stable (see Table 2). Pearson correlations were calculated between all the variables included from the NIHPD dataset. (3) External clinical validation: The NPI rule was applied to two novel datasets from a previous study (Yeates et al., Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) of children known to have higher rates of neuropsychological impairment than normative or community control samples: a control group of children with orthopedic injuries (OI) and children who sustained mild/moderate or severe traumatic brain injury (TBI). The sample characteristics for these groups are presented in Table 3. In comparison to the NIHPD sample, children from the orthopedic control group were not screened for pre-existing learning or attentional problems (Yeates et al., Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002), had significantly lower levels of intellectual functioning (t=3.84; p<.001), were from families with lower levels of education (χ2=68.80; p<.001), and were more often of minority status (white vs other race, χ2=11.91; p<.001) than those in the NIHPD sample. Ten variables were chosen to mirror the variables and eight domains used from the NIHPD dataset (see Table 2). As the choice of measures differs substantially between research projects and in clinical practice, the goal of this external validation was to test the performance of the rule when different tests are used and to assess the performance of the rule in populations suspected to exhibit higher levels of impairment. Rates of impairment were then calculated using the NPI rule for the OI and TBI groups. Pearson correlations were calculated between all the variables selected from the Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) study.
Table 3. Characteristics of the samples from Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002).
a IQ level based on performance on the Wechsler Intelligence Scale for Children Third Edition (WISC-III, Wechsler, 1991)
b Education level categories were defined as Graduate degree, College=1–4 years of college education, High school=high school graduate or fewer years of education.
c N=45 severe TBI and 53 mild/moderate TBI; lowest GCS for TBI group: M=9.79 (4.9), range 3–15.
Results and Rule Derivation
The results for the NIHPD participants on each of the neuropsychological tests included in the derivation dataset are presented according to age group in Table 4, and correlations between variables included in the NIHPD dataset (all versions) are presented in Table 5. The number of individuals with impairments on any of the eight measures (indicators of impairment) in the original dataset is presented in Table 6 for each age group and the total sample as a function of cutoff score.
Table 4. Mean performance (SD) of individuals in the NIHPD sample on the eight neuropsychological measures used in the initial derivation by age
Table 5. Pearson correlations between tests from the NIHPD database
CANTAB=Cambridge Neuropsychological Test Automated Battery; CVLT=California Verbal Learning Test; FSIQ=Full Scale IQ; ID/ED Intra-Extradimensional Set-shifting; NEPSY=Developmental Neuropsychological Assessment; Purdue=Purdue Pegboard; WASI=Wechsler Abbreviated Scale of Intelligence; WJ=Woodcock-Johnson Tests of Cognitive Abilities.
*p<.05.
**p<.01.
Table 6. Distribution (percentages) of individuals in the NIHPD study with impairments on any of the eight neuropsychological measures (i.e., number of indicators of impairment) at three levels of functioning: 1, 1.5, 2 standard deviations below the mean by age group
Note. Shading represents the number of individuals considered for the NPI rule as having impairments (1.5 SD below the mean) on two or more indicators.
NPI Rule Derivation
Indicators of impairment on each of the individual measures were considered at three cutoff scores, i.e., 1, 1.5, or 2 SDs below the mean. Frequency distributions are presented for all three levels, as there is not universal agreement on cutoffs for the interpretation of cognitive impairment. Higher cutoff scores have improved sensitivity for detecting cognitive problems, but have reduced specificity (Brooks & Iverson, Reference Brooks and Iverson2012). A relatively liberal definition of impairment was first considered by the research group using a −1 SD cutoff. Although this cutoff may be ill-suited for defining impairment in normative populations because it would rarely be considered as clinically meaningful in a typically developing child, it may be relevant to use in settings where researchers or clinicians seek to identify more subtle deviations from average performance or where the accumulation of minor difficulties may impact on functioning. A more conservative definition was considered by using a −2 SD cutoff. Although this cutoff for defining impairment has been used in clinical and research settings, it is considered to be a striking deficit and strict cutoff. Arguably, performances at 1.5 SD below the average may also be clinically and functionally significant.
In the NIHPD sample, the −2 SD cutoff lead to a very conservative interpretation of performance, with only 0.5% of the sample showing a deficit on two or more subtests and 8.9% showing a deficit on one or more subtests (see Table 6). The −1.5 SD cutoff offered a less restrictive definition of impairment, with 5.1% of individuals defined as impaired on two or more subtests and 24.1% on one or more subtests . The −1 SD cutoff identified a larger number of typically developing children as being impaired, with 25.5% of children impaired on two or more subtests and 53.8% on one or more subtests.
A further consideration in deriving a rule for neuropsychological impairment is the number of indicators of impairment that are necessary. The results in Table 6 indicate that very few individuals had three or more indicators of impairment, regardless of the cutoff used. At a −1 SD cutoff, using three indicators would identify 9.8% of the population as impaired, while using a −1.5 SD cutoff would identify impairment in only 0.8% of the sample. No individuals would be identified as impaired when using three indicators at a −2 SD cutoff. Most individuals had no deficits on any subtests or only on one subtest for that cutoff.
Thus, based on the frequency distributions presented in Table 6, the definition of neuropsychological impairment that best fits the NIHPD data and identifies approximately 95% of the population as “typically developing” is the following: “A neuropsychological impairment is present when an individual performs 1.5 standard deviations below the mean on two or more measures.” Our derived NPI rule identifies 5.1% of the total sample as impaired on two or more of the eight subtests in the assessment battery, which covers six domains of neuropsychological functioning. Applying this rule to individual age groups identifies between 3.0 and 7.2% of the population as impaired, suggesting that the case definition is appropriate for children between the ages of 6 and 18 years.
Working with this definition, we sought to explore which tests and domains of functioning were most frequently impaired and whether there was any systematic pattern of impairment (see Table 7). The highest percentages of impairment were found on the CANTAB ID/ED (6.5%) and the CVLT (6.1%), while the lowest percentage was found on the WASI FSIQ, with only one individual showing a deficit (0.3%). When the proportion of individuals impaired on each test was compared (see Table 8), most tests did not have a significantly different proportion of individuals with impairment. However, six tests had a significantly higher proportion of impairment compared to the WASI FSIQ. As noted previously, the NIHPD sample had above average intellectual functioning. This may have contributed to the difference between the IQ measure and other subtests.
Table 7. Frequency of impairment in the NIHPD sample by subtest and domain
Table 8. Holm-Bonferroni adjusted p-values for McNemar tests comparing the proportion of individuals impaired between neuropsycholgical tests
* Significant differences in the proportion of individuals impaired between tests.
Associations with Income and Education
Significant negative associations were found between number of indicators of neuropsychological impairment and parental level of education (χ2(1,368)=9.19; p=.002) and income (χ2(1,366)=12.97; p<.001) (see Table 9), with those children from families with greater income and higher levels of education demonstrating fewer indicators of neuropsychological impairment. When the number of indicators of neuropsychological impairment were broken down by level of education, NPI rates were significantly higher for the group with parental education of high school or less (11.1%) than those with college (4.4%) or graduate (3.5%) education. The college and graduate groups were therefore combined and a significant difference was also found when the “college or above” and “high school” groups were compared on number of indicators of impairment, χ2(1,368)=11.22; p=.001. However, the magnitude of these associations was small, as denoted by weak spearman rank correlations (income ρ=−.15, education ρ=−.13). Nonetheless, an education-adjusted cutoff was explored in the current sample resulting in a −1.4 SD cutoff for college and −1.6 SD cutoff for high school.
Table 9. Number of indicators of impairment at 1.5 SD below the mean, by mothers’ education level and income
NPI Rule Validation
Bootstrap validation
A bootstrap analysis with 1000 samples was performed to validate the rule. The bootstrap estimate of the proportion below the 1.5 SD criterion was 5.1% (the same as the observed estimate). The 95% bootstrap confidence interval for this proportion was 3.0 to 7.6%. This suggests that, had the NIHPD data been different (but sampled from the same population), the estimated proportion identified as impaired using our rule could have been as low as 3.0% or as high as 7.6%.
Internal and external clinical validation
The number of individuals with impairments on neuropsychological measures (indicators of impairment) is presented in Table 10 for each validation database (NIHPD/Yeates et al., Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) and for each version (10 variables/16 variables) as a function of the 1.5 SD below the mean cutoff. When the NPI rule (performance 1.5 SDs below the mean on two or more subtests) was applied to a more comprehensive version of the NIHPD data that included 8 domains and 10 variables (intellectual functioning removed, language, visuo-constructive, and academic functions added), the rule identified 6.2% of the total sample as impaired. The rate of impairment rose slightly to 7.8% in the 16 variable dataset; however, neither of these changes was significantly different from the proportion of individuals identified as impaired in the original rule (5.1%, 10 variable version z=−.64; p=.52; 16 variable version z=−.86; p=.39).
Table 10. Testing the NPI: Distribution (percentages) of individuals in the NIHPD and Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) studies with impairments on the neuropsychological measures (i.e., number of indicators of impairment) at 1.5 standard deviations below the mean
Note. Shading represents the number of individuals considered for the NPI rule as having impairments (1.5 SD below the mean) on two or more indicators.
TBI=traumatic brain injury.
When the NPI rule was applied to a novel dataset of children with orthopedic controls and children with TBI (10 variables, 8 domains), the rule identified 17.7% and 25.5% of the samples as impaired, respectively. Correlations between variables included in the Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) study are presented in Table 11. When rates of impairment were compared across TBI severity levels, 16/45 (35.6%) of children with severe TBI were impaired according to our definition compared to 9/53 (17.0%) of those with mild/moderate injuries, a statistically significant difference (z=−2.10; p=.04). The rate of impairment in children with mild/moderate TBI was comparable to the orthopedic control sample (z=−.01; p=.91).
Table 11. Pearson correlations between tests from the Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) database (orthopedic injury group)
CELF=Clinical Evaluation of Language Fundamentals; CNT=Contingency Naming Test ; CPT=Conners Performance Test; CVLT=California Verbal Learning Test; COWA=Controlled Oral Word Association Test; WISC=Wechsler Intelligence Scale for Children-III; WJ=Woodcock-Johnson Tests of Cognitive Abilities.
*p<.05.
**p<.01.
Discussion
We derived a clinical case definition rule for defining neurocognitive impairment using a large representative U.S. sample of children 6 to 18 years who completed a comprehensive neuropsychological battery. After considering a variety of cutoff scores and number of indicators of impairment, we concluded that the best rule for identifying neurocognitive impairment in 5% of the population would require two or more performances 1.5 SDs below the mean when using a test battery covering six domains of functioning and eight subtests (IQ, attention/working memory, memory, executive functions, processing speed, fine motor skills).
The cutoff used for the rule derivation results in 95% of the population being defined as typically developing (therefore, a 5% rate of identifying cognitive impairment). While this cutoff is somewhat arbitrary, it was chosen to establish a relatively conservative NPI definition. Some may prefer a less conservative definition, but this would increase the identification rate of abnormal performances in presumed healthy children. This rule is in accord with the principles outlined by Brooks and Iverson (Reference Brooks and Iverson2012). That is, many individuals had low scores on at least one of the eight subtests and this proportion decreased when a more conservative cutoff was used. No individuals in the current sample had deficits on more than six of the eight subtests.
There was no clear pattern of subtests or domains that contributed more or less to the NPI rule. However, lower rates of impairment were found on the IQ estimate compared to measures of memory, executive functions, processing speed, and fine motor skills. While these differences are likely to be in part related to the fact that the level of intellectual functioning in the NIHPD sample was higher than expected norms, they may also reflect the relative insensitivity of IQ to variations in performance in normative groups and some clinical samples (e.g., traumatic brain injury, Crowe, Catroppa, Babl, & Anderson, Reference Crowe, Catroppa, Babl and Anderson2012; Ewing-Cobbs, Barnes, & Fletcher, Reference Ewing-Cobbs, Barnes and Fletcher2003).
The number of indicators of impairment was significantly associated with parental income and education level. In the NIHPD study, income was also found to be significantly associated with neuropsychological outcome. However, when its predictive ability was tested, the association was significant for broad indictors such as general academic achievement (e.g., reading comprehension and calculation) and IQ subtests, but specific neurocognitive measures were not significantly related to income (e.g., working memory, verbal recall) (Waber et al., Reference Waber, De Moor, Forbes, Almli, Botteron, Leonard and Group2007). There is literature to suggest that more low scores are found in individuals with fewer years of parent education (Schoenberg, Lange, Brickell, & Saklofske, Reference Schoenberg, Lange, Brickell and Saklofske2007; van der Sluis, Willemse, Boomsma, & Posthuma, Reference van der Sluis, Willemsen, de Geus, Boomsma and Posthuma2008). This means the impairment identification rate increases with fewer years of education in parents and decreases in those with parents with more education. Our NPI rule is validated by the expected association found between education level and indicators of impairment, a finding that is also consistent with recent literature describing the effects of socioeconomic status (SES) on brain and cognitive development, suggesting that SES affects neuropsychological functioning and that higher rates of impairment should be found in children from lower SES homes (see, Hackman, Farah, & Meaney, Reference Hackman, Farah and Meaney2014). Despite this, the magnitude of the association between the number of indicators of impairment and education was weak. While those wishing to apply a more conservative definition of impairment that takes parental education into consideration may want to consider an education-adjusted rule, doing so does not result in particularly meaningful cutoff scores (i.e., 1.4 SD cutoff for college and −1.6 SD cutoff for high school level education) and is technically and clinically problematic.
The performance of the NPI rule was tested when domains and variables are increased. First, the rule was applied to an alternate version of the NIHPD dataset. Full scale IQ was removed because it was the only variable that had a very low proportion of impairment (only one individual) and because it consists of several different functions and is therefore more heavily weighted. Additional domains of functioning including language and visuo-spatial construction were added to verify the performance of the rule for a more comprehensive neuropsychological battery. Doing so resulted in a small increase in the rate of impairment identified (from 5.1% to 6.2%), suggesting that the rule is relatively robust to increases in the number of indicators. We expected that further increasing the number of subtests would result in an even greater number of deficits; however, a 16 variable version was associated with only a modest increase in the impairment rate (7.8%), which remained comparable to the range of impairment detected in the rule derivation when age groups were considered (3.0–7.2%). It is probable that raising the number of subtests further would result in additional increases in the number of deficits on the test battery (Brooks & Iverson, Reference Brooks and Iverson2012). This should be considered when applying the rule to even longer test batteries. If the number of subtests or variables were considerably greater than 16, the NPI derivation approach outlined here could be applied to examine impairment distributions, provided adequate standardized norms or a normative group is available.
The NPI rule was also validated by applying it to novel datasets with comparable domains of functioning. The rule was applied to the performance of two groups of children, those with OI and those who sustained TBI (Yeates et al., Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002). As expected, doing so resulted in an increase in the rate of impairment in the OI group (10 variable NIHPD data set 6.2% vs. 10 variable OI dataset 17.7% rate of impairment). This change suggests that the NPI rule is sensitive to changes in sample characteristics, such as disparities in race and SES, and differences in cognitive functioning. Children with OI also typically present with greater pre-morbid intellectual, attentional and learning difficulties than children from the community, and as such are thought to represent a good match for children with brain injuries who have similar baseline characteristics (Babikian et al., Reference Babikian, Satz, Zaucha, Light, Lewis and Asarnow2011; Yeates, Reference Yeates2010).
Logically, the rate of impairment was even higher (25.5%) when the rule was applied to a clinical population of children with TBI, which included severe injuries known to produce neuropsychological deficits post-injury (Babikian & Asarnow, Reference Babikian and Asarnow2009; Beauchamp & Anderson, Reference Beauchamp and Anderson2013), further demonstrating the sensitivity of the rule to changes in cognitive performance. It is noteworthy that although 17% of the children in the OI group were impaired according to the NPI rule, the majority of these children (82%) had impairments on only two indicators and only two individuals (18%) had impairments on four or five indicators. In contrast, the majority of the TBI group had more appreciable rates of impairment (60% impaired on three or more indicators). Of interest, children with mild/moderate injuries had a rate of impairment comparable to that of the orthopedic control group (17.0% and 17.7%, respectively), while the severely injured group had a 35.6% rate of impairment. This finding accords with a previous publication that showed equivalent rates of neuropsychological impairment in the OI and mild/moderate TBI groups over the first 4 years post-injury (Fay et al., Reference Fay, Yeates, Wade, Drotar, Stancin and Taylor2009).
Although the NPI rule was derived based on the specific tests included in the NIHPD study and not on a priori considerations of test battery choice, they provide a reasonable semblance of the type of neuropsychological assessment carried out both in clinical and research settings, as they cover six neurocognitive domains and include eight measures in a streamlined battery. The measures used in the NIHPD study are for the most part standardized and well-validated tests that are known to neuropsychologists. Only the two subtests from the CANTAB were not standardized, but were from a well-validated cognitive test battery thought to be useful for identifying brain-behavior correlations (Waber et al., Reference Waber, De Moor, Forbes, Almli, Botteron, Leonard and Group2007). Domains that were not included in the original derivation, but are typical of neuropsychological assessment, include language, academic skills, and visuo-spatial functioning (Brooks & Iverson, Reference Brooks and Iverson2012). These domains were, however, included in an alternate version of the data set, and the NPI rule was shown to perform similarly when these changes were made. Neuropsychologists often use questionnaires to obtain parental or self-reports on behavioral adjustment and social competence, but these were not included in the case definition because they are not direct measures of performance and are subject to social desirability and parental bias. Social cognition is an area of assessment that was also absent from the test battery, as there were no sociocognitive measures included at the time the NIHPD study was conducted. The availability of standardized, validated measures for the pediatric population is still limited in this domain, but is becoming increasingly relevant to both clinical and research assessments (Beauchamp & Anderson, Reference Beauchamp and Anderson2010).
Limitations
Our rule derivation is somewhat limited by the sample population and databases used. First, the NIHPD study was conducted in the United States and may not be representative of the ethnic and socioeconomic distribution of other countries, although it is reported to be an adequate reflection of the 2000 U.S. census (Waber et al., Reference Waber, De Moor, Forbes, Almli, Botteron, Leonard and Group2007). Demographic details are reported and can be compared to local characteristics of other groups and clinical samples. Second, the level of intellectual functioning in the NIHPD study population was above the normative mean. Waber and colleagues (Reference Waber, De Moor, Forbes, Almli, Botteron, Leonard and Group2007) suggest that the higher scores obtained in their sample are due to the fact that sources of morbidity were screened out by the exclusionary criteria. This is likely to be an adequate reflection of the level of functioning in typical comparison and control groups in research studies, who are also subject to medical and developmental exclusion criteria and tend to come from average to above average functioning families. Consistent with expectations, a higher rate of impairment was identified when the rule was applied to an orthopedically injured group of children who were not screened on the basis of pre-morbid cognitive difficulties and had a lower mean level of IQ, lower SES, and were more likely to be from minority groups. Third, a further limitation is that we were bound by the measures selected as part of the NIHPD and Yeates et al. (Reference Yeates, Taylor, Wade, Drotar, Stancin and Minich2002) studies. As a result, a combination of local and population norms were used in the rule derivation and validation because some tests were not standardized. Fourth, the rule’s applicability to the performance of children under 6 years of age is unclear. When three age groups spanning 6 to 18 years were considered, the same definition of clinical impairment held true for all ages, with the exception that it was derived based on seven rather than eight measures in children 6–8 years of age. The rapid cognitive development that occurs in the preschool years and the substantial difference in neuropsychological tests used in younger age groups may require the derivation of a case definition rule specific to children under 6 years (Diamond, Reference Diamond2000; Isquith, Crawford, Espy, & Gioia, Reference Isquith, Crawford, Espy and Gioia2005; Lezak, Reference Lezak1995; Tsujimoto, Reference Tsujimoto2008). Finally, the rule does not take into account pre-morbid levels of functioning, but this reflects the typical situation of clinical interpretations, as baseline data is seldom available other than third party report on academic performance (e.g., grades).
Conclusion
The NPI rule is based on psychometrically derived base rates of neurocognitive functioning in typically developing children and adolescents. The rule is expected to be useful to scientists and clinicians attempting to determine whether particular individuals or clinical populations demonstrate overall neuropsychological impairment when typical domains of neuropsychological functioning are assessed. Future studies should aim to prospectively validate the NPI rule in other normative populations and cross-validate in additional clinical groups. Other work could compare its utility in identifying neuropsychological impairment to other approaches, such as the Global Deficit Score method proposed by Carey and colleagues (Reference Carey, Woods, Gonzalez, Conover, Marcotte, Grant and Heaton2004). Further refinement of the rule may consider the addition of younger age groups, examination of the performance of the rule in terms of abnormal scatter of scores rather than number of low scores, exploration of supplementary domains such as somatosensory functioning and social cognition, and different ethnic, and socioeconomic distributions.
Acknowledgments
This work was supported by the Canadian Institutes of Health Research (R.Z. 293380), (M.B. 231125). We would like to thank Dr. Gerry Taylor for his willingness to provide the external validation data and for facilitating access to it. B.L.B. reports that he receives royalties for a pediatric memory test (Sherman and Brooks, 2015, Child and Adolescent Memory Profile [ChAMPTM], PAR Inc.), past research support from a computerized test publisher (CNS Vital Signs) through in-kind test credits, and book royalties from Oxford University Press. None of the other authors report any conflicts of interest. Data used in the preparation of this article were obtained from the Pediatric MRI Data Repository created by the NIH MRI Study of Normal Brain Development. This is a multi-site, longitudinal study of typically developing children, from ages newborn through young adulthood, conducted by the Brain Development Cooperative Group and supported by the National Institute of Child Health and Human Development, the National Institute on Drug Abuse, the National Institute of Mental Health, and the National Institute of Neurological Disorders and Stroke (Contract #s N01-HD02-3343, N01-MH9-0002, and N01-NS-9-2314, -2315, -2316, -2317, -2319, and -2320). A listing of the participating sites and a complete listing of the study investigators can be found at http://www.bic.mni.mcgill.ca/nihpd/info/participating_centers.html.
