In their book Vigor, Shadmehr and Ahmed synthesize the vast literature on movement and reward to compose an intriguing thesis: that the kinematics of our movements betray how much we value the goal we move toward. Their core argument is that – at the expense of more effort – we increase movement speed and decrease movement latency to more quickly reach goals that are valuable to us. With a pen for poetry, they craft beautiful examples of vigor and sloth in movements that could fill the pages of a popular science book. But the authors take the reader on a different route, deeper into the material, meticulously building a coherent case for their thesis. This is a scientific magnum opus of the kind that is rarely seen these days, by two outstanding scientists in neuroscience and biomechanics, taking a passionate look at the relation between movement and reward through the spectacles of economics. In the first part of their book, they introduce optimal foraging theory as a mathematical framework for their argument, and review the evidence for its quantitative, empirically testable predictions regarding the link between reward and vigor. In the second part of the book, they focus on eye movements, in particular – the motor system that has been studied the most – recasting the classic literature of the neural control of saccades from a neuroeconomic perspective. Weaving together different fields of investigation, their analysis makes a strong case for the proposed link between reward and movement vigor, geared toward maximizing what is known as the global capture rate (i.e., the rewards gained less the efforts spent, in a given time). This link inspires behavioral, electrophysiological, and neurochemical research questions and, more often than not, the authors' predictions across these various levels of analysis are met.
Here, we consider a fundamental consequence of Shadmehr and Ahmed's thesis that remained unexplored throughout their book. In an active agent, movement is wedded to perception more than to any other function of the brain. Saccadic eye movements – the type of movement that the authors focus on throughout the bulk of their book – are a prime example of this bond. Small, fixational eye movements allow the visual system to code space by time (Rucci, Ahissar, & Burr, Reference Rucci, Ahissar and Burr2018). Saccades bring the fovea to different parts of the visual scene (Rayner, Reference Rayner2009). And large-scale gaze shifts, involving movement of the head and the trunk, bring new parts of the world into view (Land, Reference Land2004). These movements do not only change what we look at, they impose rapid global displacements of the visual scene on the retina that require keeping track of where things are in space (Cavanagh, Hunt, Afraz, & Rolfs, Reference Cavanagh, Hunt, Afraz and Rolfs2010; Wurtz, Reference Wurtz2008). Every degree of change in movement vigor, thus, entails an equal degree of change in the perceptual consequences of these movements. Shadmehr and Ahmed briefly allude to this idea, arguing that “we are blind for a total of 1.5 hours during each waking day, making it particularly important for the brain to optimize duration of each saccade” (p. xii). They calculate that higher movement speeds (as those observed in response to high reward) could reduce this time by 10 min a day, which would yield a gain of many months of seeing over a human lifetime. Although these numbers are flabbergasting, this argument might not age well. We now know that the processing of visual information acquired strictly during a saccade is intact and functional, serving object continuity across saccades and facilitating gaze correction upon saccade landing (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Thus, reducing the duration of saccade-induced blindness might not be a top priority of this sensorimotor system.
Our point, therefore, is a different one. Imagine you would get a chip implanted that optimized your movement skills – including what is commonly called “muscle memory” and the reward-based mechanisms maintaining speed and accuracy. You would be gifted, say, with the nimble movements of a Parkour master, the rapid dexterity of an E-sports champion, or a professional dancer's finesse in combining intricate body movements. Would you instantly run, play, or dance at their level of skill? We argue that you would not. Just like tuning the engine of a car for higher speed would impose stress on other key parts of the machine (say, the steering, the suspension, or the brakes), an isolated update of the motor and reward system might leave the other core systems of the brain incapable to catch up. This point is most evident for perception – when we alter the way we move, new sensory information arrives at a different rate. More vigorous (i.e., high-speed and low-latency) eye movements, for instance, would require more efficient visual processing, as the sensory consequences of saccades hit the visual system at a quicker pace. Although the speed of looking at a desirable object – such as the candy bar spotlighted by the authors – may reveal its subjective value, the fast movement does not achieve its goal if the visual system is not prepared to apprehend the next part of the scene.
Thus, the efficiency of a sensory system must match the vigor of the movement system that alters its input. To achieve and maintain an appropriate, systemic balance, there appears to be a need for plasticity in perceptual and attentional processes in response to changes in movement vigor. In spelling out this argument, the focus can remain on saccadic eye movements, as they showcase our point prominently. Saccades are visual actions that are inextricably linked to their sensory consequences – every movement of the eyes across the visual scene yields an immediate, equal and opposite movement of the scene across the retina. The perceptual consequences of saccades depend on saccadic peak velocity (Ostendorf, Fischer, Finke, & Ploner, Reference Ostendorf, Fischer, Finke and Ploner2007) and the timing of post-saccadic visual information (Balsdon, Schweitzer, Watson, & Rolfs, Reference Balsdon, Schweitzer, Watson and Rolfs2018; Castet, Jeanjean, & Masson, Reference Castet, Jeanjean and Masson2002). In addition, even though pre- and intra-saccadic stimuli are routinely omitted from conscious perception (Campbell & Wurtz, Reference Campbell and Wurtz1978; Duyck, Collins, & Wexler, Reference Duyck, Collins and Wexler2016), visual processing remains effective during omission (Watson & Krekelberg, Reference Watson and Krekelberg2009) and serves fundamental visuomotor functions (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Changes in vigor should thus have immediate consequences for visual processing during and around the time of saccades. Is the visual system prepared to deal with these consequences?
Next to nothing is known about the plasticity of intra-saccadic visual processes (for a first exception, see Scholes, McGraw, & Roach, Reference Scholes, McGraw and Roach2021), so we will focus on another key player in the active visual system – attentional selection. Predictive attentional processes support vision across saccades (Rolfs, Reference Rolfs2015). Some 100 ms before the eyes move to a new location, the part of the scene that the saccade aims for stands out from the background (Rolfs & Carrasco, Reference Rolfs and Carrasco2012) and can be more easily discerned than other locations in the scene (Deubel & Schneider, Reference Deubel and Schneider1996; Ohl, Kuper, & Rolfs, Reference Ohl, Kuper and Rolfs2017). As a consequence of the saccade, objects that have a fixed place in the world rapidly shift to new positions on the retina (e.g., the target of a saccade shifts to the fovea). Pre-saccadic attention enhances performance in a broad range of spatial frequencies, with an emphasis on the highest spatial frequencies that can be resolved (at a given eccentricity), presumably to prepare for foveal processing (Kroell & Rolfs, Reference Kroell and Rolfs2021). At the same time, sensory tuning toward features of the target object sharpens as movement preparation progresses (Li, Barbot, & Carrasco, Reference Li, Barbot and Carrasco2016; Ohl et al., Reference Ohl, Kuper and Rolfs2017). To keep track of attended objects' changing locations, visual processing relies on the predictive updating of this pre-saccadic attention (Rolfs, Jonikaitis, Deubel, & Cavanagh, Reference Rolfs, Jonikaitis, Deubel and Cavanagh2011), and we have argued that it is this concert of attentional processes that gives rise to perceptual continuity across saccades (Cavanagh et al., Reference Cavanagh, Hunt, Afraz and Rolfs2010; Rolfs, Reference Rolfs2015; Rolfs & Szinte, Reference Rolfs and Szinte2016). To maintain perceptual continuity in the face of changes in saccadic vigor, therefore, we need to understand how differences in movement latency and velocity could alter the dynamics of these predictive attentional processes.
We suggest that the pace of information arriving in volleys on the retina itself would allow the visual system to sharpen its attentional priorities in time and space, and increase sensitivity to the features of objects it is going to look at next. One way to achieve that would rely on the same quantity that, in Shadmehr and Ahmed's view, controls movement vigor – reward, harvested at each new fixation. Although the direct impact of reward on pre-saccadic attention (as to our knowledge) has never been investigated, reward does alter visual processing in a way similar to pre-saccadic attention. For instance, reward history strongly shapes attentional selection in favor of high-reward stimuli with corresponding advantages in visual processing (Failing & Theeuwes, Reference Failing and Theeuwes2018). Higher reward improves orientation discrimination by sharpening behavioral orientation tuning functions (Baldassi & Simoncini, Reference Baldassi and Simoncini2011), and increases visual sensitivity in response to exogenous spatial cueing (Engelmann & Pessoa, Reference Engelmann and Pessoa2007). Correspondingly, stimuli associated with high reward elicit stronger responses in visual cortex and in the attentional control network than low-reward stimuli do (Serences, Reference Serences2008). Indeed, selection based on reward history as opposed to selective attention seems to be hard to dissociate at a neural level. Curiously, the basal ganglia – a key player in the modulation of vigor according to Shadmehr and Ahmed – are involved in the control of visual attention (Arcizet & Krauzlis, Reference Arcizet and Krauzlis2018). They may thus constitute a shared origin of simultaneous changes in reward-related priorities for action as well as attentional influences on perception.
The similarities between the consequences of reward and pre-saccadic attention for visual processing suggest that the mechanisms to prepare the visual system for faster (or slower) movements are in place, in that reward serves both the motor and the visual system at the same time. Although the relation between vigor and visual processing remains largely unexplored, some evidence suggests that pursuing a research program in this direction could be fruitful.
First, there is at least one hint that pre-saccadic attention shifts are malleable, and that this plasticity occurs as a consequence of implicit reward. White, Rolfs, and Carrasco (Reference White, Rolfs and Carrasco2013) had observers saccade to one of six patches of moving dots. The target location, and the motion direction of each patch, were randomly chosen on each trial. Observers were asked to execute the saccade and discriminate a brief luminance pulse (the probe) displayed some time before movement onset at an unpredictable location. They showed that, just before saccade onset, performance in the luminance discrimination task (their proxy for the deployment of selective attention) was tied specifically to the saccade target location. Interestingly, this spatial specificity was reduced when the probe had appeared at a non-target location on the previous trial. Another way to put this result is that observers paid more attention to non-target locations when they had just made the experience that the perceptual task was spatially dissociated from the saccade target. Along the same vein, if on the previous trial, the motion direction at the probed location matched that of the saccade target, then the current target's motion direction improved performance across all locations. Thus, the recent history of utility of the feature and location of the saccade target was associated with adaptive changes in pre-saccadic attention.
Second, Jonikaitis and colleagues showed that attention shifts more vigorously to the target of a saccade when the imminent movement has a shorter as compared to a longer latency (Jonikaitis & Deubel, Reference Jonikaitis and Deubel2011). This pattern of results was consistent across many data sets (Jonikaitis & Theeuwes, Reference Jonikaitis and Theeuwes2013) and suggests that attention is coupled to movement onset, not to the onset of the instruction to move. A more intriguing aspect of this result, however, is that the dynamics of predictive attentional processes may covary with the vigor of our movements.
Such links between movement and attention provide a rare glimpse at how visual processing covaries with the kinematics of movement control (for another striking example in the domain of perception, see van Heusden, Rolfs, Cavanagh, & Hogendoorn, Reference van Heusden, Rolfs, Cavanagh and Hogendoorn2018). Future research should address directly how changes in movement vigor accelerate or decelerate perceptual processes, and how, at the same time, the needs of the visual system may impose constraints on the variability and plasticity of movement vigor.
In their book Vigor, Shadmehr and Ahmed synthesize the vast literature on movement and reward to compose an intriguing thesis: that the kinematics of our movements betray how much we value the goal we move toward. Their core argument is that – at the expense of more effort – we increase movement speed and decrease movement latency to more quickly reach goals that are valuable to us. With a pen for poetry, they craft beautiful examples of vigor and sloth in movements that could fill the pages of a popular science book. But the authors take the reader on a different route, deeper into the material, meticulously building a coherent case for their thesis. This is a scientific magnum opus of the kind that is rarely seen these days, by two outstanding scientists in neuroscience and biomechanics, taking a passionate look at the relation between movement and reward through the spectacles of economics. In the first part of their book, they introduce optimal foraging theory as a mathematical framework for their argument, and review the evidence for its quantitative, empirically testable predictions regarding the link between reward and vigor. In the second part of the book, they focus on eye movements, in particular – the motor system that has been studied the most – recasting the classic literature of the neural control of saccades from a neuroeconomic perspective. Weaving together different fields of investigation, their analysis makes a strong case for the proposed link between reward and movement vigor, geared toward maximizing what is known as the global capture rate (i.e., the rewards gained less the efforts spent, in a given time). This link inspires behavioral, electrophysiological, and neurochemical research questions and, more often than not, the authors' predictions across these various levels of analysis are met.
Here, we consider a fundamental consequence of Shadmehr and Ahmed's thesis that remained unexplored throughout their book. In an active agent, movement is wedded to perception more than to any other function of the brain. Saccadic eye movements – the type of movement that the authors focus on throughout the bulk of their book – are a prime example of this bond. Small, fixational eye movements allow the visual system to code space by time (Rucci, Ahissar, & Burr, Reference Rucci, Ahissar and Burr2018). Saccades bring the fovea to different parts of the visual scene (Rayner, Reference Rayner2009). And large-scale gaze shifts, involving movement of the head and the trunk, bring new parts of the world into view (Land, Reference Land2004). These movements do not only change what we look at, they impose rapid global displacements of the visual scene on the retina that require keeping track of where things are in space (Cavanagh, Hunt, Afraz, & Rolfs, Reference Cavanagh, Hunt, Afraz and Rolfs2010; Wurtz, Reference Wurtz2008). Every degree of change in movement vigor, thus, entails an equal degree of change in the perceptual consequences of these movements. Shadmehr and Ahmed briefly allude to this idea, arguing that “we are blind for a total of 1.5 hours during each waking day, making it particularly important for the brain to optimize duration of each saccade” (p. xii). They calculate that higher movement speeds (as those observed in response to high reward) could reduce this time by 10 min a day, which would yield a gain of many months of seeing over a human lifetime. Although these numbers are flabbergasting, this argument might not age well. We now know that the processing of visual information acquired strictly during a saccade is intact and functional, serving object continuity across saccades and facilitating gaze correction upon saccade landing (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Thus, reducing the duration of saccade-induced blindness might not be a top priority of this sensorimotor system.
Our point, therefore, is a different one. Imagine you would get a chip implanted that optimized your movement skills – including what is commonly called “muscle memory” and the reward-based mechanisms maintaining speed and accuracy. You would be gifted, say, with the nimble movements of a Parkour master, the rapid dexterity of an E-sports champion, or a professional dancer's finesse in combining intricate body movements. Would you instantly run, play, or dance at their level of skill? We argue that you would not. Just like tuning the engine of a car for higher speed would impose stress on other key parts of the machine (say, the steering, the suspension, or the brakes), an isolated update of the motor and reward system might leave the other core systems of the brain incapable to catch up. This point is most evident for perception – when we alter the way we move, new sensory information arrives at a different rate. More vigorous (i.e., high-speed and low-latency) eye movements, for instance, would require more efficient visual processing, as the sensory consequences of saccades hit the visual system at a quicker pace. Although the speed of looking at a desirable object – such as the candy bar spotlighted by the authors – may reveal its subjective value, the fast movement does not achieve its goal if the visual system is not prepared to apprehend the next part of the scene.
Thus, the efficiency of a sensory system must match the vigor of the movement system that alters its input. To achieve and maintain an appropriate, systemic balance, there appears to be a need for plasticity in perceptual and attentional processes in response to changes in movement vigor. In spelling out this argument, the focus can remain on saccadic eye movements, as they showcase our point prominently. Saccades are visual actions that are inextricably linked to their sensory consequences – every movement of the eyes across the visual scene yields an immediate, equal and opposite movement of the scene across the retina. The perceptual consequences of saccades depend on saccadic peak velocity (Ostendorf, Fischer, Finke, & Ploner, Reference Ostendorf, Fischer, Finke and Ploner2007) and the timing of post-saccadic visual information (Balsdon, Schweitzer, Watson, & Rolfs, Reference Balsdon, Schweitzer, Watson and Rolfs2018; Castet, Jeanjean, & Masson, Reference Castet, Jeanjean and Masson2002). In addition, even though pre- and intra-saccadic stimuli are routinely omitted from conscious perception (Campbell & Wurtz, Reference Campbell and Wurtz1978; Duyck, Collins, & Wexler, Reference Duyck, Collins and Wexler2016), visual processing remains effective during omission (Watson & Krekelberg, Reference Watson and Krekelberg2009) and serves fundamental visuomotor functions (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Changes in vigor should thus have immediate consequences for visual processing during and around the time of saccades. Is the visual system prepared to deal with these consequences?
Next to nothing is known about the plasticity of intra-saccadic visual processes (for a first exception, see Scholes, McGraw, & Roach, Reference Scholes, McGraw and Roach2021), so we will focus on another key player in the active visual system – attentional selection. Predictive attentional processes support vision across saccades (Rolfs, Reference Rolfs2015). Some 100 ms before the eyes move to a new location, the part of the scene that the saccade aims for stands out from the background (Rolfs & Carrasco, Reference Rolfs and Carrasco2012) and can be more easily discerned than other locations in the scene (Deubel & Schneider, Reference Deubel and Schneider1996; Ohl, Kuper, & Rolfs, Reference Ohl, Kuper and Rolfs2017). As a consequence of the saccade, objects that have a fixed place in the world rapidly shift to new positions on the retina (e.g., the target of a saccade shifts to the fovea). Pre-saccadic attention enhances performance in a broad range of spatial frequencies, with an emphasis on the highest spatial frequencies that can be resolved (at a given eccentricity), presumably to prepare for foveal processing (Kroell & Rolfs, Reference Kroell and Rolfs2021). At the same time, sensory tuning toward features of the target object sharpens as movement preparation progresses (Li, Barbot, & Carrasco, Reference Li, Barbot and Carrasco2016; Ohl et al., Reference Ohl, Kuper and Rolfs2017). To keep track of attended objects' changing locations, visual processing relies on the predictive updating of this pre-saccadic attention (Rolfs, Jonikaitis, Deubel, & Cavanagh, Reference Rolfs, Jonikaitis, Deubel and Cavanagh2011), and we have argued that it is this concert of attentional processes that gives rise to perceptual continuity across saccades (Cavanagh et al., Reference Cavanagh, Hunt, Afraz and Rolfs2010; Rolfs, Reference Rolfs2015; Rolfs & Szinte, Reference Rolfs and Szinte2016). To maintain perceptual continuity in the face of changes in saccadic vigor, therefore, we need to understand how differences in movement latency and velocity could alter the dynamics of these predictive attentional processes.
We suggest that the pace of information arriving in volleys on the retina itself would allow the visual system to sharpen its attentional priorities in time and space, and increase sensitivity to the features of objects it is going to look at next. One way to achieve that would rely on the same quantity that, in Shadmehr and Ahmed's view, controls movement vigor – reward, harvested at each new fixation. Although the direct impact of reward on pre-saccadic attention (as to our knowledge) has never been investigated, reward does alter visual processing in a way similar to pre-saccadic attention. For instance, reward history strongly shapes attentional selection in favor of high-reward stimuli with corresponding advantages in visual processing (Failing & Theeuwes, Reference Failing and Theeuwes2018). Higher reward improves orientation discrimination by sharpening behavioral orientation tuning functions (Baldassi & Simoncini, Reference Baldassi and Simoncini2011), and increases visual sensitivity in response to exogenous spatial cueing (Engelmann & Pessoa, Reference Engelmann and Pessoa2007). Correspondingly, stimuli associated with high reward elicit stronger responses in visual cortex and in the attentional control network than low-reward stimuli do (Serences, Reference Serences2008). Indeed, selection based on reward history as opposed to selective attention seems to be hard to dissociate at a neural level. Curiously, the basal ganglia – a key player in the modulation of vigor according to Shadmehr and Ahmed – are involved in the control of visual attention (Arcizet & Krauzlis, Reference Arcizet and Krauzlis2018). They may thus constitute a shared origin of simultaneous changes in reward-related priorities for action as well as attentional influences on perception.
The similarities between the consequences of reward and pre-saccadic attention for visual processing suggest that the mechanisms to prepare the visual system for faster (or slower) movements are in place, in that reward serves both the motor and the visual system at the same time. Although the relation between vigor and visual processing remains largely unexplored, some evidence suggests that pursuing a research program in this direction could be fruitful.
First, there is at least one hint that pre-saccadic attention shifts are malleable, and that this plasticity occurs as a consequence of implicit reward. White, Rolfs, and Carrasco (Reference White, Rolfs and Carrasco2013) had observers saccade to one of six patches of moving dots. The target location, and the motion direction of each patch, were randomly chosen on each trial. Observers were asked to execute the saccade and discriminate a brief luminance pulse (the probe) displayed some time before movement onset at an unpredictable location. They showed that, just before saccade onset, performance in the luminance discrimination task (their proxy for the deployment of selective attention) was tied specifically to the saccade target location. Interestingly, this spatial specificity was reduced when the probe had appeared at a non-target location on the previous trial. Another way to put this result is that observers paid more attention to non-target locations when they had just made the experience that the perceptual task was spatially dissociated from the saccade target. Along the same vein, if on the previous trial, the motion direction at the probed location matched that of the saccade target, then the current target's motion direction improved performance across all locations. Thus, the recent history of utility of the feature and location of the saccade target was associated with adaptive changes in pre-saccadic attention.
Second, Jonikaitis and colleagues showed that attention shifts more vigorously to the target of a saccade when the imminent movement has a shorter as compared to a longer latency (Jonikaitis & Deubel, Reference Jonikaitis and Deubel2011). This pattern of results was consistent across many data sets (Jonikaitis & Theeuwes, Reference Jonikaitis and Theeuwes2013) and suggests that attention is coupled to movement onset, not to the onset of the instruction to move. A more intriguing aspect of this result, however, is that the dynamics of predictive attentional processes may covary with the vigor of our movements.
Such links between movement and attention provide a rare glimpse at how visual processing covaries with the kinematics of movement control (for another striking example in the domain of perception, see van Heusden, Rolfs, Cavanagh, & Hogendoorn, Reference van Heusden, Rolfs, Cavanagh and Hogendoorn2018). Future research should address directly how changes in movement vigor accelerate or decelerate perceptual processes, and how, at the same time, the needs of the visual system may impose constraints on the variability and plasticity of movement vigor.
Financial support
This study was supported by the Deutsche Forschungsgemeinschaft (MR, grant number RO3579/12-1), (SO, grant number OH274/2-2), and the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (MR, Grant agreement number 865715).
Conflict of interest
None.