INTRODUCTION
This paper addresses the issue of the provenance of clinical tests of
executive function, and how best to set about developing the next
generation of clinical measures. We will argue that whilst traditional
tests of executive function have been remarkably useful, we are now at the
stage in the development of the field where one could create bespoke tests
specifically intended for clinical applications rather than adapting
procedures emerging from purely experimental investigations, as has been
almost exclusively the case until recently. We will further argue that
whilst one should continue to consider as potentially useful these
procedures which emerge almost coincidentally, a more planned and
deliberate programme should consider a “function-led” approach
as a particularly fruitful avenue. Such an approach may have many
scientific and clinical advantages, and address recent criticisms of
existing assessment procedures (e.g., Manchester et
al., 2004). But to make these points, it will first be necessary to
define our terms:
Argument 1. The distinction between construct, cognitive
operation, and behavioural function: A taxonomy for discussion.
The term ecological validity, first coined by Brunswik (1956, pp. 48–50; see Hammond,
1966 for review), was originally used to refer to the degree of
relation between a proximal (e.g., retinal) cue and the distal (e.g.,
object) variable in perceptual experiments (see Hammond, 1998, for discussion). Now, ironically, it is
most often used in reference to quite separate concerns of
Brunswik's, those of “generalisability” and
“representativeness.” This misuse (from a historical
perspective) is now so prevalent that we will, somewhat reluctantly, adopt
the “incorrect” use of the term here. We will adapt the
nomenclature of Kvavilashvili and Ellis (2004)
for the current application by using “ecological validity” to
refer both to the degree of “representativeness” of a task
(the extent to which a clinical test corresponds in form and context to a
situation encountered outside the laboratory), and the
“generalisability” of test results (the degree to which poor
performance on the test will be predictive of problems outside the
laboratory; for recent examples of this use see Chaytor
& Schmitter-Edgecombe, 2003; Clark et al.,
2000; Coughlan et al., 2004; Farias et al., 2003; Gioia &
Isquith, 2004; Higginson et al., 2000;
Kibby et al., 1998; McDonald
et al., 2004; Odhuba et al., 2005; Semkovska et al., 2004).
The foregoing argument will also require clear distinctions between
the levels of explanation used in the cognitive neuroscience of executive
function. So we will start with a very basic description of terms. We will
make a distinction between Constructs, Operations, and
Functions (see Figure 1). (Perhaps the
most commonly used term, Processes, will not be used here since
it is a term which has been used so flexibly in the literature.)
Construct refers here (as is conventional) to a hypothetical
cognitive resource whose existence is inferred from research findings
(e.g., a correlation between two variables). Common examples of constructs
are “Working Memory,” and “General intelligence.”
These are ideas which motivate experiments, and are
interpretations of data sets rather than phenomena that are part
of them. In this respect, it is important to note, as Coltheart and Davies
(2003) point out, that “The truth of an
explanatory theory goes beyond the data and so is never logically
guaranteed by the data” (p. 188).
Levels of explanation in the cognitive neuroscience of executive
function.
Operations in the current context are the individual
component steps of cognition which are not directly observable, but are
inferred from a combination of task analysis and change in some dependent
variable. In a different circumstance this could signify the involvement
of some other operation. They may or may not be broken down into
subcomponents. Many experimental observations in the cognitive
neuroscience or neuropsychology of executive function would fall into this
category. For instance, increases in reaction time or errors, changes in
cerebral blood flow and so forth, have all been held to be indicators
(i.e., observable correlates of) of cognitive operations. It is important
to note here that operations can be understood at the level of the
individual rather than the individual's interaction with the
environment, and that this is another level at which they differ from
functions. The operation of single-word reading can happen in the
mind without any outward behavioural sign (e.g., speech), but be
accompanied perhaps by changes in blood flow in the reading areas of the
brain. It is also important to note that operations are behavioural
changes whose utility (rather than existence) can be made only in
reference to outcome in the world. In one context, the absence of a
behaviour can be a useful thing—for example, withholding a behaviour
in a motor inhibition task—but in another may be evidence of
dysfunction—say, inability to produce a word in the Hayling Sentence
Completion Task part 2 (Burgess & Shallice,
1996). Functions, by contrast, are the directly observable
behavioural output which is the product of a series of operations. They
are usually understood in terms of a goal, instruction or intention to
act, and it is therefore usually clear what constitutes success or
failure. Simple examples might be reading, naming objects, describing a
previously heard story; more complex behavioural examples might be
preparing a meal, mailing a letter at a previously intended time, and so
forth.
An example of the use of this kind of taxonomy appears in the
excellent study by Owen et al. (1999). In the
abstract to the study, the purpose of the study is described thus:
“[this] study used positron emission tomography (PET) to
demonstrate that working memory processes within the human
mid-dorsolateral and mid-ventrolateral frontal regions are organized
according to the type of processing required…. Two spatial working
memory tasks were used which varied in the extent to which they required
different executive processes…. During a ‘spatial span’
task that required the subject to hold a sequence of five previously
remembered locations in working memory a significant change in blood-flow
was observed in the right mid-ventrolateral frontal cortex…. By
contrast, during a ‘2-back’ task that required the subject to
continually update and manipulate an ongoing sequence of locations within
working memory, significant blood flow increases were observed in both
mid-ventrolateral and mid-dorsolateral frontal regions (Owen et al., 1999, p. 567). It is clear from this
description (and the rest of the paper) that the operations of
“holding a sequence of numbers” in mind, and continually
updating and manipulating an ongoing sequence are considered potentially
fractionable, but both fall within the general cognitive domain of
“working memory” (a construct).
Argument 2. The study of “central processes”
makes special demands: From theory to practice in the cognitive
neuroscience of executive function.
Burgess (1997) argues that the field of
executive function (EF) differs from most others in cognitive neuroscience
in the differing emphasis on these levels of explanation. Specifically,
the use of constructs in the field of executive function is much more
prevalent. This is largely because of the “low process-behaviour
correspondence” (pp. 85–89) inherent in the study of
“central processes” compared with other fields which deal with
more “routine processing” or “encapsulated
resources” (Shallice, 1988; both terms
assuming a particular organisation of the cognitive system). The abilities
supported by the frontal lobes are thought to operate largely on the
products of processing of other systems, and also output through other
systems. Thus the operation of the “central executive” (Baddeley, 1996) or “supervisory attentional
system” (Shallice, 1988) that the frontal
lobes are thought to support is bound up with the operation of other
systems to a degree that the nonexecutive components of, say, vision or
hearing are not. Anatomical support, for instance, comes from the absence
of direct pathways between prefrontal cortex and motor cortex (Leichnetz, 1986; Passingham,
1993, p. 125). And the clearest supporting demonstration is that,
unlike with other cerebral regions, destruction of prefrontal cortex is
not associated with loss of basic sensory or motor capacities or with
obvious impairment of speech (Benton, 1991, p.
3). Linked to this is that at least some of the processing supported by
prefrontal structures has been shown to be material-nonspecific (Burgess et al., 2001; 2003;
Gilbert et al., 2005). In other words, it
operates on input from a variety of sources, with a corresponding lack of
dedicated process-output pathways.
This contrasts sharply with conceptions of fields of study dealing
with behaviour deriving from the operation of abilities which are supposed
to be more dedicated to it. As an example of an “encapsulated
resources” theory, Burgess (1997)
considers Bruce and Young's (1986) model of
face recognition (where the behaviour of seeing a face and identifying
would be a function, and the theoretical individual contributory
components to this behaviour—such as recognition—are the
operations). And although more recently, the debate concerning
how people identify faces has taken a more complex stance, contrasting
these “domain-specific” accounts with
“domain-general” ones, where visual recognition proceeds
through the operation of resources which are specialised for a particular
form of processing that can operate on a wide variety of classes of
stimuli (Duchaine et al., 2003, 2004; Kanwisher, 2000), the
processes are still assumed to be specialised for the processing of
visual stimuli. This general principle could be extended to most
other areas of enquiry within cognitive neuroscience, such as visual
perception, reading, spoken language, motor control, audition, and so
forth, where the predominant theories all hold that there are resources
dedicated to that domain of cognition, if not the modality (see, e.g.,
MacSweeney et al., 2004, for an example within
the auditory linguistic processing domain) or at least are agnostic as
regards this possibility. By contrast, according to most conceptions,
executive processes theoretically manifest themselves in a range
of quite different situations, involving quite different stimuli, the only
unifying feature of which might be the involvement of that process.
To illustrate the universality of this viewpoint, we will consider
briefly the opinion of six leading theories of executive function. The
most obvious example of a theory of the “central processing”
kind is Duncan's (e.g., Duncan, 2005; Duncan et al., 1995) contention that “executive
processes” are synonymous with Spearman's g, and his
recent “adaptive coding” model is a clever theory extending
this assertion by addressing the possible neural underpinning of
g (e.g., Duncan & Miller, 2002).
However, there are many other theories which share this domain-nonspecific
aspect (see, e.g., Burgess & Alderman, 2004;
Burgess & Robertson, 2002 for brief review).
For instance the theory of Cohen and colleagues (e.g., Cohen & Servan-Schreiber, 1992; Cohen et al.,
1990, 1998) holds that
their single system may underpin both active memory and behavioural
inhibition, determined by the current context, and they specifically state
that their own theory is incomplete as regards the representation of many
different information types, and this aspect is left as a lacuna in the
model.
This aspect of “central processing” (i.e., domain- and
material-insensitive) is also reflected in “working memory”
theories of executive function. For example, Petrides (1994, 1998; Petrides & Pandya, 2002) maintains that the
mid-ventrolateral PFC “subserves the expression within memory of
various first-order executive processes, such as (the) active selection,
comparison and judgement of stimuli,” (p. 107) without specifying
further what those stimuli might be. Goldman-Rakic's (1995) working memory account does however propose some
basic domain- or material-specificity since the key aspect of it is that
the various different frontal lobe regions all perform a similar role in
working memory, but that each processes a different type of information.
However, the account still holds that different forms of observable
behaviour may result from dysfunction in one central processing resource:
Working memory is defined as the ability to “hold an item of
information ‘in mind’ for a short period of time and to update
information from moment to moment,” (Goldman-Rakic, 1998, p. 90) and it is argued that
dysfunction of this system can cause a variety of deficits (see Goldman-Rakic & Leung, 2002, for further
discussion.).
Even those theories that propose a series of potentially dissociable
executive resources still maintain this centrality (e.g., Fuster 1997; 2002), but this
characteristic is most obvious in those executive accounts which rest upon
the concept of “attention.” For example, Stuss et al. (1995) propose seven different attentional functions:
sustaining, concentrating, sharing, suppressing, switching, preparing, and
setting. But since there is no specification of the specific situations or
material which require the differential use of these resources, the reader
is invited to assume that they operate regardless of the form or source of
the input source. Even the most complex model in this domain, that of
Shallice (e.g., Norman & Shallice, 1986;
Shallice, 1988, 2002;
Shallice & Burgess, 1991a, 1991b, 1993, 1996; Shallice et al., 1989)
still utilises the concept of attention, where the possibility of domain-
or material-specificity is either denied or left as a lacuna. Most
recently, the putative organisation of the SAS has been articulated in
more detail (Shallice & Burgess,
1996/1998). In this model, the SAS plays a part in at least
eight different processes: working memory, monitoring, rejection of
schema, spontaneous schema generation, adoption of processing mode, goal
setting, delayed intention marker realisation, and episodic memory
retrieval. However it is largely assumed that these central processes are
independent of modality (of input or output). (For further evidence see
also computational accounts, e.g., Dehaene et al.,
1998, and representational accounts, e.g., Grafman, 2002.)
Argument 3. “Construct hegemony”: Historical
hangover?
The field of executive function has been dominated by these
construct-level, “macro” theories for the last 20 years in
particular. By contrast there are relatively few accounts of the
processing required to deal with specific situations which might tap these
constructs. The extent to which this is the case may in fact in time
become of interest to historians of science. For instance, one of the
earliest and most compelling descriptions of cognitive deficit following
frontal lobe damage was given by Penfield and Evans (1935; see also studies by Ackerly
& Benton, 1947; Brickner, 1936). This
seminal paper described a catastrophic impairment in the higher-order
organisation of behaviour, manifesting itself for instance particularly
sharply as an inability to coordinate the various activities involved in
cooking a meal (see also Fortin et al., 2003).
Given this observation, one might imagine that by now we would know quite
a lot about the processes involved in cooking a meal, since Penfield and
Evan's astute and influential observation in the context of a
relative dearth (at that time) of empirical evidence would surely have
suggested a “cardinal” situation for tapping the cognitive
processes supported by the frontal lobes, and opened up a promising avenue
for empirical studies. But this is not the case at all. We know virtually
nothing about how the brain allows us to organise simple, everyday
activities like cooking, and interacts with environmental factors in doing
so (although there are a few honourable exceptions, see, e.g., Kirsh, 1995). Instead, EF research has spent several
decades investigating the dynamics of (by comparison) esoteric activities
such as performing the Wisconsin Card Sorting Test.
So how and why did EF research develop in this way? This may at least
in part be a consequence of the history of research in the field of
executive/frontal lobe function. The very existence of cognitive
abilities supported by the frontal lobes was doubted for many years (Benton, 1991, p.19). Indeed, whilst early experimental
physiologists were making sound progress in the fields of visual
perception and language more than a hundred years ago, frontal lobe
function was beset with null findings and failures to replicate effects
across species. The scepticism this engendered was repeated in the
observation of humans with frontal lobe damage: Even the facts of what was
to become probably the most famous case of the “frontal lobe
syndrome” (Harlow's [1848; 1868] Phineas Gage) were not believed by some at
the time of its report. This attitude persisted well into the 1960s: For
example, Teuber (1964), one of the leading
authorities of the time, referred to the functions of the frontal lobes as
a “riddle.” In retrospect this lack of development in the area
seems odd: The reports of the effects of frontal lobe damage in humans
were actually quite consistent, and there also many successful
demonstrations in primates (e.g., Jacobsen,
1931; 1936). Moreover, the effects of
damage to frontal lobe and related structures was considered predictable
enough for frontal lobe psychosurgery to thrive (e.g., Malmo, 1948) during this period. The scepticism
persisted, however, and one might argue that the tendency of investigators
to assign ill-defined labels to the behaviours they saw (e.g.,
“perseveration”), rather than investigate the characteristics
of the often myriad situations that provoked them, meant that research
became quite conservative in its method (i.e., tending to use the same
tasks and paradigms), whilst also—in the absence of constraining
empirical evidence—rather ambitious in the scope of its
theorising.
However, the past 25 years have seen rapid development in the
cognitive neuroscience of executive functions, prompted in particular by
technological advance (e.g., new methods of brain imaging, better methods
of localising lesions). The repeated and sometimes unexpected findings of
prefrontal involvement in even the simplest tasks has overthrown the old
view of the frontal lobes as “functionally silent,” and now no
leading theoretician of whom we are aware would maintain that the frontal
lobes do not support “executive” (Pribram,
1973) abilities that are critical to human cognition.
Such a rapid development has however not been without cost, and it
could be argued that the rate of innovation of method in the field of
executive function has not matched the rate of technological advance. The
majority of the early research papers using the new technologies (e.g.,
functional brain imaging) understandably leaned towards using the
previously existing tools, principally (for example), because of the need
to validate new technologies. This may have had the unintended effect of
prolonging the use of these older tests and procedures both experimentally
and clinically. Moreover, the rapid swing (in historical terms) from one
extreme position (i.e., the frontal lobes are functionally silent, ergo
“executive functions” don't exist) to the other (i.e.,
the frontal lobes support executive processing which is involved in a very
wide range of situations [see, e.g., Burgess et al., 2005b, 2006; Grady, 1999 for reviews]) has invited explanation
at the construct level. Moving beyond this level is a challenge: The
empirical findings in the field of executive functions of course support
the newer view at the broadest level, but are still beset with
complications. For example, Stuss et al. (2000,
2002) have shown that relatively small changes
in task format can have marked differences in terms of demands upon
executive abilities (see also Burgess, 1997, for
review of methodological difficulties), and the sheer volume of recent
findings presents its own conceptual difficulties. For example, in trying
to understand the functions of the largest single cytoarchitectonic PFC
region (Brodmann's Area [BA] 10), findings of BA 10 PFC
blood flow changes (see, e.g., Gilbert et al., in
press; Grady, 1999) in a very wide range
of types of tasks in imaging studies provides few constraints for
theorising (see Burgess et al., 2005b for
discussion).
The combination of these interrelated factors—historical
inheritance, rapid conceptual change; numerous but often complex and
unpredictable empirical findings—plus the practical and technical
difficulties of investigations at the function level has, we would
contend, invited accounts of executive function at the construct
level, and for this reason basic scientific investigations (i.e.,
contrasted with applied or clinical investigations) have been, until very
recently, overwhelmingly driven by this perspective.
This construct-level, or construct-to-operation-level theorising,
combined with the experimental conservatism engendered by the historical
research difficulties has led to research into executive abilities largely
concentrating upon these two facets (i.e., examination of particular
experimental, and clinical, paradigms on the one hand, and construct-level
theorising of the type above on the other) whilst largely ignoring the
issue of the translation of the hypothesised resources into the real
world. The dynamic interplay between situational factors and the
hypothesised resources (i.e., function-level analysis) has
received very little attention. This has led to gaps in the links between
the three explanatory levels which are required in order to link
explanations at the basic science level with those required in clinical
practice (i.e., my hypothetical resource can be termed Alpha; it supports
cognitive operations X, Y and Z; and these resources are tapped by, or
used to deal with, situations which have characteristics A, B, and C). Yet
it is precisely at this level that clinical tests of executive function
should in large part be targeted since in clinical practice the most
important issue is usually performance outside the testing situation. How
odd that one should not try to mimic (or at least understand) the demands
of life outside the clinic when trying to determine the generalisability
of test performance within it! Instead a common practice in clinical work
and applied research has been to use performance on tests which
emerged—often many years ago—from traditional experimental
psychological investigations. The standpoint of these original studies has
largely been to view the variability in contextual or environmental
factors which naturally occur outside the lab as a confound (see Kvavilashvili & Ellis, 2004, for discussion of the
long history of similar issues within the memory research domain). But it
is exactly at this level (i.e., the interaction between the individual and
their context: the “function” level) in which the clinician is
interested. This dominance of the construct level investigation in
neuroscience executive function research is reflected in (or probably
reflects) the historical focus of the related field of cognitive science
on the formation and manipulation of internal representations, rather than
the ways in which agents interact with their environments in order to
achieve goals (see, e.g., Clark, 1997; Hutchins, 1995).
Argument 4. Time to consider a new approach?
One can add to these criticisms that traditional tests were not
developed to address the concerns of clinicians—that is, to measure
the most clinically significant deficits and do so in a way that makes the
implications of the test score quite plain.
Consider for example, the history of, and current application of the
currently most used and most thoroughly investigated single measure in the
field of human frontal lobe function: the Wisconsin Card Sorting Test
(WCST). This test was developed (in almost its current form) by Esta Berg
in 1948, and slightly modified by Grant and Berg (1948). Both these studies involved only healthy
participants, and the procedure was merely a variant of those used much
earlier. In fact, the origin of the association between sorting tasks and
frontal lobe function can be traced to Gelb and Goldstein (1920), who first developed sorting procedures for
assessment of brain damage following the First World War. They adapted for
clinical use the methods that had been used by Ach and his colleagues to
investigate concept formation in healthy individuals. Following Gelb and
Goldstein's discoveries, there were many investigations of sorting
impairments in various patients groups, including those with brain-damage,
dementia, learning difficulties, and schizophrenia carried out by groups
in Germany and Russia (see Weigl, 1927; Goldstein & Scheerer, 1941 for review). The
translation into English of a number of these papers during the
1935–45 period was followed by a rapid increase of interest in these
methods in the U.S.A. Unfortunately, the reluctance of the American
authors of the time to cite the original papers rather than their
translations gives the appearance that these later papers were more
contemporary than they were.
It is an important historical irony that by the time the WCST was
developed, there were already tried and tested (by the standards of the
time) sorting procedures available for clinical use. Yet it is the WCST,
which was not developed for clinical use, that has now become dominant.
This is largely attributable to the effect of a study by Milner (1963) who, more than 40 years since the original
demonstration of sorting impairments, administered the Grant & Berg
procedure to a group of neurological patients, with virtually the only
modification being to give the test twice in succession.
Milner's principal achievement was to show that patients with
dorsolateral prefrontal lesions performed worse on the test than those
with either orbitofrontal or lesions outside the frontal lobes. However
this finding has never to our knowledge been entirely replicated, either
by Milner or by any other investigator, although there are many failures
to replicate (see, e.g., Anderson et al., 1995;
Corcoran & Upton, 1993; van den Broek et al., 1993). Psychometric studies have
now shown that the WCST is a multifactorial test (e.g., Greve et al., 1997; Miyake et al.,
2000). These results are congruent with those from recent
functional imaging studies that show a network of frontal and nonfrontal
brain regions to be involved in WCST performance (e.g., Marenco et al., 1993; Mentzel et
al., 1998), and similar results from careful human lesion work
(Stuss et al., 2000). Both methodologies stress
how relatively minor changes in task format can have marked effect upon
results.
Now consider in this context the typical uses of the WCST in clinical
assessment. It is possible to give a rationale for understanding the three
levels of WCST based on current common conceptions thus: WCST is a measure
of “working memory” (construct), which supports
“concept shifting” (operation), which enables correct
choice of card sorting (function). However, the explanation at
the level of construct is not very helpful in clinical assessment at the
individual level, since such explanations go beyond the data and thus
belong in the field of scientific hypothesis-testing rather than patient
diagnosis. And the outline of the empirical findings above suggests
explanation at the level of operation is not well-established (e.g., there
may be many different reasons for failure, especially in different
populations). Finally—and this is a particularly critical
problem—the situation that performance of the WCST presents is so
unlike everyday situations, that knowledge of performance in it is of very
little help for assessment since it is of uncertain predictive validity
(or “generalisability”): We do not really know what situations
in everyday life require the abilities that the WCST measures.
Further examples of the problem
This situation is largely repeated for most other popular measures
used to investigate executive (i.e., “frontal lobe”) function.
For example, the Stroop task was invented in 1935 (Stroop, 1935), but not applied in frontal lobe
research until 1974 (Perret, 1974). Perret
showed that patients with left frontal lobe damage were impaired in the
interference condition of the task. Since that time the only partial
replication of the result in humans to our knowledge was given recently by
Stuss et al. (2001; “partial”, since
Stuss et al. showed a left frontal deficit for colour naming not
interference!), although the link between test performance and the frontal
lobes now also seems secure on the basis of functional imaging results. It
again seems likely from the results of the Stuss et al. study, and
failures to replicate (see, e.g., Shallice,
1982) that the precise format of the task is an important factor in
determining results. A similar picture emerges for the Tower of London
test and its variants. This test was invented by Shallice and McCarthy
(Shallice, 1982) as an application of AI theory
in cognitive neuroscience. It has been generally considered to be a
measure of “planning.” However, it is now doubtful that the
reasons for task failure in frontal lobe patients could be characterised
as a planning impairment (e.g., Goel & Grafman,
1995; Morris et al., 1997; see Burgess et al., 2005a for review). Moreover, the task
is multicomponential (Carlin et al., 2000; Hodgson et al., 2000), psychometrically complex (Welsh
et al., 1999, 2000)
and the frontal lobe contribution to the task may be better characterised
as mental imagery (Rowe et al., 2001) and goal
conflict resolution (Morris et al., 1997) rather
than planning-specific functions. Moreover, what the Tower of London
measures may in any case have little relevance to “real-world”
planning (Burgess et al., 2005a; Goel et al., 1997). For most of the established tests
of executive function, for example, Trail-Making, Cattell's
Culture-Fair test, similar arguments can be made In short, these
procedures share two characteristics: First, they were not originally
designed for the purpose to which they are currently put, and second, the
dynamics of the tasks have been shown to be complex.
Argument 5. Moving forward: What are the ideal
requirements of a suitable task?
However, it is a third characteristic that makes these points even
more significant for this area: The situation that test performance in the
clinic presents to the participant is quite unlike most situations which
will be encountered outside it (i.e., they show poor
“representativeness”). Experimental investigations of basic
cognitive systems often use tasks that are simplified “models of the
world” to make the problem scientifically tractable. For example, it
is easy to define the circumstances where the processes underlying visual
motion detection or colour vision operate. Similarly, those working in the
field of motor control often use “models of the world”
deliberately designed to have characteristics close to those encountered
in everyday life (see, e.g., Haggard & Richardson,
1996; Ramnani et al., 2001). However it
is not at all clear which “models of the world” are
represented by tasks such as the WCST. Certainly current conceptions of
them are far removed from their origins. Moreover, it is not at all clear
that they are simplified models of the world at all. With the
WCST, it is widely acknowledged that there is a minimum of three types of
failure, which may interact in complex ways. Why has experimentation not
favoured tasks that are more obviously like real-world situations?
Argument 6: “Ecologically valid” tests can be
psychometrically as sound as experimentally derived ones.
We have already argued that the history of research in this area and
the interconnected factor of construct-led hypothesising has contributed
to a dearth of investigation at the function level. But a further
contributing problem has probably been the widely held assumption that
tasks very close to real world situations will be psychometrically
unsound. Recently, this has been shown not to be the case (e.g., Burgess et al., 1998; Knight et
al., 2002; Wilson et al., 1998). Indeed,
ironically perhaps, tasks that are obviously “models of the
world” (e.g., Alderman et al.'s [1996] Zoo-Map Test or Shallice &
Burgess's [1991a] Multiple Errands task) generally return
psychometric values higher than many traditional neuropsychological
measures (e.g., free recall of short stories, or meaningless figures;
Coughlan & Hollows, 1985; Knight et al., 2002). And even formalised
“real-world” activities such as the Multiple Errands Test
(MET) can show surprisingly good psychometric characteristics. The MET is
at one extreme of the “representativeness” dimension, being a
“real-life” multitasking test carried out in shopping
precinct. Participants have to complete a number of tasks whilst following
a set of rules (e.g., no shop should be entered other than to buy
something). The tasks vary in terms of complexity (e.g., buy a small brown
loaf vs. discover a currency exchange rate), and there are a
number of “hidden” problems in the tasks that have to be
appreciated and the possible course of action evaluated (e.g., one item
asks that participants write and send a postcard, yet they are given no
pen). In this way, the task is, like many situations outside the
laboratory, quite “open-ended” or “ill-structured”
(Goel & Grafman, 2000). There are many
possible courses of action, and it is up to the individual to choose one.
Knight et al. (2002) report inter-rater
reliabilities ranging between .81 to 1.00 for the various measures of the
MET, and an internal consistency (Cronbach's alpha) of .77. Moreover,
Dawson et al. (2005b) report inter-class
correlation coefficients (measuring inter-rater reliability) on a hospital
version of the Multiple Errands Test in 30 controls and 30 people with
acquired brain injury of ≥ 0.90 (p ≤ 0.01).
Furthermore, and perhaps more important from a clinical standpoint,
“real-world model” tasks may have particularly good
“generalisability”—that is, that performance in the task
situation is predictive of performance in situations outside it (e.g.,
Fortin et al., 2003). For example, Dawson et al.
(2005a) have shown that Multiple Errands
performance (using a hospital-based version) not only correlates well in
CVA patients with self-report measures of everyday ability (the Sickness
Impact Profile [Bergner et al., 1981];
the Mayo-Portland Adaptability Inventory [Malec
& Lezak, 2003]), but also with objective assessment of
daily living skills (using the Assessment of Motor and Process Skills
[AMPS] battery, e.g., Fisher, 2003).
Despite the relatively small group size (CVA patients, n = 11;
healthy controls n = 11), Dawson et al. nevertheless showed
correlations between MET and the AMPS-M scale which reached significance
at p < .05 (−.63; the Process scale-MET correlation was
−.51). In a further study, Dawson et al. (2005b) administered the MET-HV to 30 people with
acquired brain injury (and 30 matched controls) and found correlations
between the AMPS and the Mayo-Portland scale of between .5 and .8, also
suggesting good generalisability.
Critically, however, Knight et al. (2002)
also report correlations between carers' ratings of everyday life
problems and traditional tests of executive function (Cognitive Estimates
[Shallice & Evans, 1978]; Modified
Wisconsin Card-Sorting [MWCST; Nelson,
1976]; Tower of London Test [Shallice,
1982]; Verbal Fluency [Miller,
1984]) as well as a version of the Multiple Errands Test
adapted for use in a typical hospital environment. Of the traditional
executive tasks, only the MWCST perseverative errors and Cognitive
Estimates tests showed any significant relation with carers' ratings
of dysexecutive symptom clusters (using the DEX; Burgess et al., 1996a). The highest of these
correlations was .47 (p = .038; between Cognitive Estimates
performance and problems with “intentionality,” such as
distractibility). However the correlation of MET-HV and ratings of
intentionality problems was .70 (p = .001). Thus these results
add to the evidence that performance on more “ecologically
valid” tasks can show closer concordance with observed symptoms in
everyday life than performance on traditional experimentally derived tasks
(Burgess et al., 1998), and can be adapted to a
clinical setting. Finally, performance on tests with high
“representativeness” can be more predictive of performance on
traditional tasks than are the traditional tasks themselves (Alderman et al., 2003; Burgess et
al., 1998; Knight et al., 2002; Wilson et al., 1998).
Argument 7: In clinical work, one cannot ignore the
necessity of knowing how the abilities one is measuring translate into
behaviour outside the laboratory. In most circumstances this should also
be the case in experimental research.
A final case for the rejection of the old procedures is that they
require the introduction of an unnecessary extra stage in brain research.
For instance, after 40 years of study of the WCST, and 20 years of Tower
of London task study, we do know a little more about the dynamics of these
tasks and what they measure. But it is perfectly possible that they
measure a form of epiphenomenon which actually has very little
significance for understanding the function of the organ under study.
Using these tasks for proxies for measurement of real-world disability
therefore requires that the operations involved in performing, say, the
WCST must be established empirically as a second stage of investigation.
This requirement is greatly diminished for measures which are
self-evidently formalised models of real-life demands.
Argument 8. The requirements of research paradigms and
clinical tools differ.
The foregoing argument should not be understood as a strong criticism
of procedures such as the WCST in basic science research, where they have
proved very useful in providing empirical evidence upon which theorising
can be based. The problem is that the purpose of tests invented for use in
either clinical or experimental contexts differ. Experimental tests
usually seek to test a specific hypothesis about brain-behaviour
relationships with the aim of elucidating the role of particular brain
regions in cognition, or building information processing models of the
cognition system that the brain supports. There is usually a conception of
what the task measures (and how it is performed). These conceptions
routinely do not stand the test of time: This is at the heart of
scientific progress. However, the practicing clinician trying to assess
and understand the nature of his/her clients' difficulties does
not enjoy the same luxury of fallability without consequence. The root of
this discrepancy is that scientific hypotheses may go beyond the data, but
clinical diagnoses should not.
Argument 9: Where might the next generation of tests come
from?
Considering these factors (and those outlined above), it is not
perhaps surprising if current tests derived directly from experimental
science are not ideal for clinical purposes. Moreover, the origin of these
tasks is perhaps more opportunistic than might be supposed, taking four
main forms: (1) Animal work, where the leap to human clinical
neuropsychological applications has, with a few arguable exceptions (e.g.,
Chorover & Cole, 1966) not been extensive.
(2) Human work, where clinical observations of the effects of ablations
have been turned into formalised tasks which, virtually unchanged, have
then been used clinically (e.g., Wisconsin Card Sorting Test, Tower of
London test; Verbal Fluency, etc.). (3) More recently, the observation of
frontal lobe haemodynamic changes has then either used these tasks (or
similar ones), or else studied a particular operation (e.g., “memory
retrieval”) and incidentally found prefrontal involvement. However
to our knowledge this research has not (yet) led to the development of a
novel clinical neuropsychological test. (4) There has also been more
conceptual research into a hypothetical construct (e.g., “working
memory”) which is putatively strongly linked to prefrontal function;
this work has yielded a few potentially useful clinical tasks (e.g., Baddeley et al., 1997).
Which of these investigative approaches is likely to provide new and
better clinical neuropsychological tests of executive function? It is
clear that (1) is unlikely to; this field has been in existence longer
than any other, and yet yielded few widely used tasks (although the
scientific advances it has yielded have been useful in less direct ways).
Approach (3) may in time yield useful tasks, but it is unclear at the
moment whether this work is necessarily relevant (e.g., there is no
necessary reason for correspondence between localisation evidence across
human lesion work and functional neuroimaging). This leaves (2) and (4).
However, tasks derived from method (4) will need to be validated, as with
every psychometric measure, against some criterion. What should this
criterion be? Observer ratings? Perhaps. But more objectively, perhaps
measurement of ability in the real-world. This will require the
development of real-world-like validation tasks. But if we are going that
far, why not consider first the use these real-world analogues as the
assessment measure rather than their experimentally-derived proxies? A
similar argument can be made for (2): If we are to develop tests which are
specifically designed for clinical use (rather than adapt experimental
paradigms of uncertain validity) surely the most obvious starting point is
to consider function-level measurement of abilities expressed as
competence in the real world.
Argument 10: Function-led research is a highly promising
avenue for the development of new clinical tests.
Interestingly, Kingstone and colleagues, investigating attention in
healthy people, have reached a similar position (e.g., Kingstone et al.,
2003, 2005). They ask,
“to what extent does the simple, impoverished, and highly artificial
experimental task … have to do with the many complex, rich, real
life experiences that people share?” (p. 344). They go on to argue
that laboratory research (in the field of attention) is based on two
critical assumptions, both of which are problematic. The first is that the
process of interest is “stable across different situations,”
in other words that processes studied in the lab are assumed to be the
same as the processes that are expressed in the real world. The second
assumption is that one can maximise analytical power by minimizing the
variability in a situation save for the factor being manipulated. In
regard to the first assumption, Kingstone et al. (2005) suggest that it “eliminates any need or
obligation by the scientist to confirm that the factors being manipulated
and measured in the lab actually express themselves in the real
world” (p. 346). As regards the second assumption, Kingstone et al.
point out that “important characteristics of complex nonlinear
systems such as the human attention system…are only revealed, or
emerge, when several variables vary together in highly specific ways (see
Ward, 2002; Weinberg,
1975)” (p. 348). (This latter point is especially germane to
the following discussion about the abilities supported by rostral PFC
processes, since one theory of the role of this region in human cognition
suggests that it is involved when two or more other processes have to be
coordinated: Ramnani & Owen, 2004.) Overall,
Kingstone et al. (2005) conclude that their work
is now oriented towards the goal of observing and describing behaviour as
it occurs: an approach they term “cognitive ethology,” away
from “making causal claims about fundamental processes.” In
terms of the framework here, they are moving away from a construct-based
approach to their research towards a function-driven one. If our analysis
above is correct, their basic science programme is of the sort which would
be most likely to yield paradigms which can be well adapted for clinical
neuropsychological use.
WHAT IS FUNCTION-LED RESEARCH? AN EXAMPLE
AND CONCLUSION
Thus far, we have explored the position that traditional tests of
executive function are unlikely (unless almost by coincidence) to be ideal
for clinical purposes. And we have argued that function-led research is
the most obvious candidate approach when considering where the new
generation of clinical assessment tools might come from. But what does
this approach actually entail? One example comes from our own lab, where
we started investigations into how the brain supports multitasking (a
function) by first considering patients who showed behavioural
disorganisation in everyday life (Shallice &
Burgess, 1991a; 1991b; see also Eslinger & Damasio, 1985). The first step was to
develop a task that was as close as feasible to a formalised version of
the “real-world” situations in which the patients had
problems: the Multiple Errands Test described above (see Figure 2).
Panel A shows a schematic representation of a typical control
performance on Shallice and Burgess's (1991a) Multiple Errands Test. Panel B shows the
impaired performance of a patient (DN) with frontal lobe damage who was
matched to the control for intellectual functioning, and who also
performed well on ten traditional clinical tests of executive function
(see Shallice & Burgess 1991a for further
details). Hatched shading indicates shops that need to be entered in order
to complete the test. Solid block shading indicates shops that do not need
to be entered, or are forbidden by the task rules. (Adapted from Burgess & Alderman, 2004.)
The second was to develop a lab-based “model of the world”
which captured a putatively critical component (voluntary multiple delayed
task-switching) of the “real-world analogue” task (Burgess, 2000a). Shallice and Burgess (1991a) showed that patients with frontal lobe damage
could be markedly impaired on these tasks despite normal performance on
traditional clinical tests of executive (and other) abilities. A third
step was to link this basic aspect of multitasking (voluntary delayed task
switching) to brain structure. This was carried out by human group lesion
studies of lab-based multitasking tests (the SET and the Greenwich Test)
which have shown that relatively circumscribed multitasking deficits
follow rostral PFC lesions (Burgess et al.,
2000; Burgess et al., submitted: described in Burgess et al., 2005b). These findings concurred with
those from studies of patients with similar problems (e.g., Bird et al., 2004; Burgess,
2000b; Goldstein et al., 1993; Levine et al., 1998; 1999;
see also Figure 3).
Brain scans of four neurological patients with rostral prefrontal
damage. All patients achieved superior scores on IQ tests, and all
achieved excellent scores on traditional executive tasks such as the
Wisconsin Card Sorting and Verbal Fluency Tests. However they all showed
significant behavioral organization problems in everyday life. Top left:
PF (Goel & Grafman, 2000); Top right: GN
(Goldstein et al., 1993); Bottom left, right: AP
and FS respectively (Shallice & Burgess,
1991a). For ease of understanding, the scans have been transposed
so that left on the scan represents the left hemisphere. Adapted from
Burgess et al. (2005b).
Then, working up the levels of explanation from function to construct,
the neural underpinnings of a key aspect of this sub-function (the
operations involved in the realisation of a delayed intention, i.e.,
prospective memory) were investigated in a series of functional imaging
experiments (Burgess et al., 2001; 2003; Okuda et al., 2004)
which confirmed a role for rostral PFC (principally BA 10) in certain
forms of prospective memory. Then on the grounds of both the lesion and
imaging studies, we hypothesised a construct underlying these various
operations: an attentional “gateway” that operates to alter
the relative valence of stimulus-oriented or stimulus-independent thought
in nonroutine circumstances, and tested the hypothesis of BA 10
involvement in a variety of indicators for such a construct in a series of
neuroimaging experiments (Gilbert et al., 2005,
2006; Simons et al., 2005a, 2005b, in press). A further step would be to investigate the
generalisability in neurological patients of the indicators for the
construct developed through this theoretical stage, thus linking
construct-level theorising with function (see Burgess et al., 2005b, 2006, for further
details of this research programme).
This argument should not be mistaken as suggesting the sole use of
tests which are carried out in the “real world” (i.e., outside
the normal lab or examination room setting) such as the MET, or that such
tests will necessarily prove more clinically useful than others,
especially when practical concerns are taken into account. Instead the
argument here is that one should consider in the design of new
tasks the demands that the “real world” may make but that are
not presented to the participant performing existing tests in a
lab setting. By not considering these aspects, we may be missing
decrements in many aspects of cognition which are critical to competence
in everyday life. Some progress has been made in various areas of
neuropsychology (e.g., Robertson et al., 1994;
Wilson et al., 2003), but with few exceptions
(e.g., Wilson et al., 1996) in the field of
executive function, this is not the provenance of the most prevalently
used tasks.
The general scientific approach to the neuroscience of executive
function advocated here (i.e., analysing the demands made by real-world
situations and then trying to mimic them in the lab) is certainly not
unique, with similar programmes occurring recently in related areas such
as social cognition (e.g. Channon et al., 2005;
Frith & Frith, 2003). If our argument is
correct, these kinds of programmes are the most likely sources for the
next generation of “executive function” clinical tests.
Certainly the one above yielded a clinical test (the Six Element Test)
which is widely used, and also a useful task with high
“representativeness” (the MET). There are now various versions
of both the Multiple Errands (e.g., Knight et al.,
2002) and the Six Element tests (e.g., Burgess
et al., 1996b; Emslie et al., 2003; Kliegel et al., 2000; Levine et
al., 2000; Manly et al., 2002), and they
have been used in the investigation of a variety of conditions (e.g.,
traumatic brain injury, Hoclet et al., 2003;
Levine et al., 2000; schizophrenia, Evans et al., 1997; depression, Channon & Green, 1999; ADHD and Oppositional
Defiant/Conduct Disorder, Clark et al.,
2000; drug abuse, Zakzanis & Young,
2001; sleep deprivation, Nilsson et al.,
2005; normal ageing, Garden et al., 2001;
“intensive care syndrome,” Sukantarat et al., in press), and there is also information about the
psychometric properties of the tests, (e.g., Alderman
et al., 2003, Jelicic et al., 2001; Kafer & Hunter, 1997; Knight et
al., 2002), their “generalisability” (Burgess et al., 1998), and their correlations with
other executive function tests (Duncan et al.,
1997).
Data taken from Alderman et al. (2003),
showing a group level dissociation between neurological patients whose
failures on the MET-SV were predominately rule-breaks, and those who
predominately failed to achieve tasks. The total patient sample was
n = 50, so these two groups together represent 90% of all
patients tested.
Moreover, the study of differences in patterns of failure on the two
tests are revealing patterns of impairment that are both theoretically
interesting and clinically relevant (see Burgess et
al., 2005b for further details). For instance, Alderman et al.
(2003) found a double-dissociation between
rule-breaking behaviour and failure to initiate tasks, and these two
patterns were associated with different dysexecutive symptoms in everyday
life. “Rule-breakers” tend to show problems with control
aspects of memory (e.g., show confabulation, temporal sequencing problems,
and repetitive behaviour). By contrast, people who fail to complete the
set tasks tend to show negative symptoms, such as apathy and shallow
affect. From a theoretical perspective, Burgess et al. (2005b) argue that these results may reflect the
“real-world” consequences of a time/event dissociation in
prospective memory. From a rehabilitation perspective the patterns suggest
different treatment applications (Burgess &
Robertson, 2002) in a much more direct way than would, say, a
dissociation between perseverative behaviour and problems with concept
shifting in performance on the WCST.
It is currently too early to tell whether the further theoretical
investigations into the constructs underpinning these functions will yield
lab-type tasks which are of clinical utility, although this seems possible
given the experiences of related fields (e.g., Channon
et al., 2005). However it seems likely that going from function to
construct presents useful constraints upon theorising which are less
apparent in going from construct to function. This approach may therefore
be useful as a starting point for both the development of new and better
clinical tests of executive function, and also for basic cognitive
neuroscience investigations.
