Hostname: page-component-745bb68f8f-mzp66 Total loading time: 0 Render date: 2025-02-06T13:19:41.234Z Has data issue: false hasContentIssue false

Moving fast and seeing slow? The visual consequences of vigorous movement

Published online by Cambridge University Press:  30 September 2021

Martin Rolfs
Affiliation:
Department of Psychology, Humboldt-Universität zu Berlin, Berlin10099, Germanymartin.rolfs@hu-berlin.de; sven.ohl@hu-berlin.de; http://www.rolfslab.de Bernstein Center for Computational Neuroscience Berlin, Berlin10115, Germany
Sven Ohl
Affiliation:
Department of Psychology, Humboldt-Universität zu Berlin, Berlin10099, Germanymartin.rolfs@hu-berlin.de; sven.ohl@hu-berlin.de; http://www.rolfslab.de

Abstract

In active agents, sensory and motor processes form an inevitable bond. This wedding is particularly striking for saccadic eye movements – the prime target of Shadmehr and Ahmed's thesis – which impose frequent changes on the retinal image. Changes in movement vigor (latency and speed), therefore, will need to be accompanied by changes in visual and attentional processes. We argue that the mechanisms that control movement vigor may also enable vision to attune to changes in movement kinematics.

Type
Open Peer Commentary
Copyright
Copyright © The Author(s), 2021. Published by Cambridge University Press

In their book Vigor, Shadmehr and Ahmed synthesize the vast literature on movement and reward to compose an intriguing thesis: that the kinematics of our movements betray how much we value the goal we move toward. Their core argument is that – at the expense of more effort – we increase movement speed and decrease movement latency to more quickly reach goals that are valuable to us. With a pen for poetry, they craft beautiful examples of vigor and sloth in movements that could fill the pages of a popular science book. But the authors take the reader on a different route, deeper into the material, meticulously building a coherent case for their thesis. This is a scientific magnum opus of the kind that is rarely seen these days, by two outstanding scientists in neuroscience and biomechanics, taking a passionate look at the relation between movement and reward through the spectacles of economics. In the first part of their book, they introduce optimal foraging theory as a mathematical framework for their argument, and review the evidence for its quantitative, empirically testable predictions regarding the link between reward and vigor. In the second part of the book, they focus on eye movements, in particular – the motor system that has been studied the most – recasting the classic literature of the neural control of saccades from a neuroeconomic perspective. Weaving together different fields of investigation, their analysis makes a strong case for the proposed link between reward and movement vigor, geared toward maximizing what is known as the global capture rate (i.e., the rewards gained less the efforts spent, in a given time). This link inspires behavioral, electrophysiological, and neurochemical research questions and, more often than not, the authors' predictions across these various levels of analysis are met.

Here, we consider a fundamental consequence of Shadmehr and Ahmed's thesis that remained unexplored throughout their book. In an active agent, movement is wedded to perception more than to any other function of the brain. Saccadic eye movements – the type of movement that the authors focus on throughout the bulk of their book – are a prime example of this bond. Small, fixational eye movements allow the visual system to code space by time (Rucci, Ahissar, & Burr, Reference Rucci, Ahissar and Burr2018). Saccades bring the fovea to different parts of the visual scene (Rayner, Reference Rayner2009). And large-scale gaze shifts, involving movement of the head and the trunk, bring new parts of the world into view (Land, Reference Land2004). These movements do not only change what we look at, they impose rapid global displacements of the visual scene on the retina that require keeping track of where things are in space (Cavanagh, Hunt, Afraz, & Rolfs, Reference Cavanagh, Hunt, Afraz and Rolfs2010; Wurtz, Reference Wurtz2008). Every degree of change in movement vigor, thus, entails an equal degree of change in the perceptual consequences of these movements. Shadmehr and Ahmed briefly allude to this idea, arguing that “we are blind for a total of 1.5 hours during each waking day, making it particularly important for the brain to optimize duration of each saccade” (p. xii). They calculate that higher movement speeds (as those observed in response to high reward) could reduce this time by 10 min a day, which would yield a gain of many months of seeing over a human lifetime. Although these numbers are flabbergasting, this argument might not age well. We now know that the processing of visual information acquired strictly during a saccade is intact and functional, serving object continuity across saccades and facilitating gaze correction upon saccade landing (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Thus, reducing the duration of saccade-induced blindness might not be a top priority of this sensorimotor system.

Our point, therefore, is a different one. Imagine you would get a chip implanted that optimized your movement skills – including what is commonly called “muscle memory” and the reward-based mechanisms maintaining speed and accuracy. You would be gifted, say, with the nimble movements of a Parkour master, the rapid dexterity of an E-sports champion, or a professional dancer's finesse in combining intricate body movements. Would you instantly run, play, or dance at their level of skill? We argue that you would not. Just like tuning the engine of a car for higher speed would impose stress on other key parts of the machine (say, the steering, the suspension, or the brakes), an isolated update of the motor and reward system might leave the other core systems of the brain incapable to catch up. This point is most evident for perception – when we alter the way we move, new sensory information arrives at a different rate. More vigorous (i.e., high-speed and low-latency) eye movements, for instance, would require more efficient visual processing, as the sensory consequences of saccades hit the visual system at a quicker pace. Although the speed of looking at a desirable object – such as the candy bar spotlighted by the authors – may reveal its subjective value, the fast movement does not achieve its goal if the visual system is not prepared to apprehend the next part of the scene.

Thus, the efficiency of a sensory system must match the vigor of the movement system that alters its input. To achieve and maintain an appropriate, systemic balance, there appears to be a need for plasticity in perceptual and attentional processes in response to changes in movement vigor. In spelling out this argument, the focus can remain on saccadic eye movements, as they showcase our point prominently. Saccades are visual actions that are inextricably linked to their sensory consequences – every movement of the eyes across the visual scene yields an immediate, equal and opposite movement of the scene across the retina. The perceptual consequences of saccades depend on saccadic peak velocity (Ostendorf, Fischer, Finke, & Ploner, Reference Ostendorf, Fischer, Finke and Ploner2007) and the timing of post-saccadic visual information (Balsdon, Schweitzer, Watson, & Rolfs, Reference Balsdon, Schweitzer, Watson and Rolfs2018; Castet, Jeanjean, & Masson, Reference Castet, Jeanjean and Masson2002). In addition, even though pre- and intra-saccadic stimuli are routinely omitted from conscious perception (Campbell & Wurtz, Reference Campbell and Wurtz1978; Duyck, Collins, & Wexler, Reference Duyck, Collins and Wexler2016), visual processing remains effective during omission (Watson & Krekelberg, Reference Watson and Krekelberg2009) and serves fundamental visuomotor functions (Schweitzer & Rolfs, Reference Schweitzer and Rolfs2020, Reference Schweitzer and Rolfsin press). Changes in vigor should thus have immediate consequences for visual processing during and around the time of saccades. Is the visual system prepared to deal with these consequences?

Next to nothing is known about the plasticity of intra-saccadic visual processes (for a first exception, see Scholes, McGraw, & Roach, Reference Scholes, McGraw and Roach2021), so we will focus on another key player in the active visual system – attentional selection. Predictive attentional processes support vision across saccades (Rolfs, Reference Rolfs2015). Some 100 ms before the eyes move to a new location, the part of the scene that the saccade aims for stands out from the background (Rolfs & Carrasco, Reference Rolfs and Carrasco2012) and can be more easily discerned than other locations in the scene (Deubel & Schneider, Reference Deubel and Schneider1996; Ohl, Kuper, & Rolfs, Reference Ohl, Kuper and Rolfs2017). As a consequence of the saccade, objects that have a fixed place in the world rapidly shift to new positions on the retina (e.g., the target of a saccade shifts to the fovea). Pre-saccadic attention enhances performance in a broad range of spatial frequencies, with an emphasis on the highest spatial frequencies that can be resolved (at a given eccentricity), presumably to prepare for foveal processing (Kroell & Rolfs, Reference Kroell and Rolfs2021). At the same time, sensory tuning toward features of the target object sharpens as movement preparation progresses (Li, Barbot, & Carrasco, Reference Li, Barbot and Carrasco2016; Ohl et al., Reference Ohl, Kuper and Rolfs2017). To keep track of attended objects' changing locations, visual processing relies on the predictive updating of this pre-saccadic attention (Rolfs, Jonikaitis, Deubel, & Cavanagh, Reference Rolfs, Jonikaitis, Deubel and Cavanagh2011), and we have argued that it is this concert of attentional processes that gives rise to perceptual continuity across saccades (Cavanagh et al., Reference Cavanagh, Hunt, Afraz and Rolfs2010; Rolfs, Reference Rolfs2015; Rolfs & Szinte, Reference Rolfs and Szinte2016). To maintain perceptual continuity in the face of changes in saccadic vigor, therefore, we need to understand how differences in movement latency and velocity could alter the dynamics of these predictive attentional processes.

We suggest that the pace of information arriving in volleys on the retina itself would allow the visual system to sharpen its attentional priorities in time and space, and increase sensitivity to the features of objects it is going to look at next. One way to achieve that would rely on the same quantity that, in Shadmehr and Ahmed's view, controls movement vigor – reward, harvested at each new fixation. Although the direct impact of reward on pre-saccadic attention (as to our knowledge) has never been investigated, reward does alter visual processing in a way similar to pre-saccadic attention. For instance, reward history strongly shapes attentional selection in favor of high-reward stimuli with corresponding advantages in visual processing (Failing & Theeuwes, Reference Failing and Theeuwes2018). Higher reward improves orientation discrimination by sharpening behavioral orientation tuning functions (Baldassi & Simoncini, Reference Baldassi and Simoncini2011), and increases visual sensitivity in response to exogenous spatial cueing (Engelmann & Pessoa, Reference Engelmann and Pessoa2007). Correspondingly, stimuli associated with high reward elicit stronger responses in visual cortex and in the attentional control network than low-reward stimuli do (Serences, Reference Serences2008). Indeed, selection based on reward history as opposed to selective attention seems to be hard to dissociate at a neural level. Curiously, the basal ganglia – a key player in the modulation of vigor according to Shadmehr and Ahmed – are involved in the control of visual attention (Arcizet & Krauzlis, Reference Arcizet and Krauzlis2018). They may thus constitute a shared origin of simultaneous changes in reward-related priorities for action as well as attentional influences on perception.

The similarities between the consequences of reward and pre-saccadic attention for visual processing suggest that the mechanisms to prepare the visual system for faster (or slower) movements are in place, in that reward serves both the motor and the visual system at the same time. Although the relation between vigor and visual processing remains largely unexplored, some evidence suggests that pursuing a research program in this direction could be fruitful.

First, there is at least one hint that pre-saccadic attention shifts are malleable, and that this plasticity occurs as a consequence of implicit reward. White, Rolfs, and Carrasco (Reference White, Rolfs and Carrasco2013) had observers saccade to one of six patches of moving dots. The target location, and the motion direction of each patch, were randomly chosen on each trial. Observers were asked to execute the saccade and discriminate a brief luminance pulse (the probe) displayed some time before movement onset at an unpredictable location. They showed that, just before saccade onset, performance in the luminance discrimination task (their proxy for the deployment of selective attention) was tied specifically to the saccade target location. Interestingly, this spatial specificity was reduced when the probe had appeared at a non-target location on the previous trial. Another way to put this result is that observers paid more attention to non-target locations when they had just made the experience that the perceptual task was spatially dissociated from the saccade target. Along the same vein, if on the previous trial, the motion direction at the probed location matched that of the saccade target, then the current target's motion direction improved performance across all locations. Thus, the recent history of utility of the feature and location of the saccade target was associated with adaptive changes in pre-saccadic attention.

Second, Jonikaitis and colleagues showed that attention shifts more vigorously to the target of a saccade when the imminent movement has a shorter as compared to a longer latency (Jonikaitis & Deubel, Reference Jonikaitis and Deubel2011). This pattern of results was consistent across many data sets (Jonikaitis & Theeuwes, Reference Jonikaitis and Theeuwes2013) and suggests that attention is coupled to movement onset, not to the onset of the instruction to move. A more intriguing aspect of this result, however, is that the dynamics of predictive attentional processes may covary with the vigor of our movements.

Such links between movement and attention provide a rare glimpse at how visual processing covaries with the kinematics of movement control (for another striking example in the domain of perception, see van Heusden, Rolfs, Cavanagh, & Hogendoorn, Reference van Heusden, Rolfs, Cavanagh and Hogendoorn2018). Future research should address directly how changes in movement vigor accelerate or decelerate perceptual processes, and how, at the same time, the needs of the visual system may impose constraints on the variability and plasticity of movement vigor.

Financial support

This study was supported by the Deutsche Forschungsgemeinschaft (MR, grant number RO3579/12-1), (SO, grant number OH274/2-2), and the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (MR, Grant agreement number 865715).

Conflict of interest

None.

References

Arcizet, F., & Krauzlis, R. J. (2018). Covert spatial selection in primate basal ganglia. PLoS Biology, 16(10), e2005930.CrossRefGoogle ScholarPubMed
Baldassi, S., & Simoncini, C. (2011). Reward sharpens orientation coding independently of attention. Frontiers in Neuroscience, 5, 13.CrossRefGoogle ScholarPubMed
Balsdon, T., Schweitzer, R., Watson, T. L., & Rolfs, M. (2018). All is not lost: Post-saccadic contributions to the perceptual omission of intra-saccadic streaks. Consciousness and Cognition, 64, 1931.CrossRefGoogle ScholarPubMed
Campbell, F. W., & Wurtz, R. H. (1978). Saccadic omission: Why we do not see a grey-out during a saccadic eye movement. Vision Research, 18(10), 12971303.CrossRefGoogle ScholarPubMed
Castet, E., Jeanjean, S., & Masson, G. S. (2002). Motion perception of saccade-induced retinal translation. Proceedings of the National Academy of Sciences, 99(23), 1515915163.CrossRefGoogle ScholarPubMed
Cavanagh, P., Hunt, A. R., Afraz, A., & Rolfs, M. (2010). Visual stability based on remapping of attention pointers. Trends in Cognitive Sciences, 14(4), 147153.CrossRefGoogle ScholarPubMed
Deubel, H., & Schneider, W. X. (1996). Saccade target selection and object recognition: Evidence for a common attentional mechanism. Vision Research, 36(12), 18271837.CrossRefGoogle ScholarPubMed
Duyck, M., Collins, T., & Wexler, M. (2016). Masking the saccadic smear. Journal of Vision, 16(10), 113.CrossRefGoogle ScholarPubMed
Engelmann, J. B., & Pessoa, L. (2007). Motivation sharpens exogenous spatial attention. Emotion (Washington, D.C.), 7(3), 668674.CrossRefGoogle ScholarPubMed
Failing, M., & Theeuwes, J. (2018). Selection history: How reward modulates selectivity of visual attention. Psychonomic Bulletin & Review, 25(2), 514538.CrossRefGoogle ScholarPubMed
Jonikaitis, D., & Deubel, H. (2011). Independent allocation of attention to eye and hand targets in coordinated eye-hand movements. Psychological Science, 22(3), 339347.CrossRefGoogle ScholarPubMed
Jonikaitis, D., & Theeuwes, J. (2013). Dissociating oculomotor contributions to spatial and feature-based selection. Journal of Neurophysiology, 110(7), 15251534.CrossRefGoogle ScholarPubMed
Kroell, L. M., & Rolfs, M. (2021). The peripheral sensitivity profile at the saccade target reshapes during saccade preparation, Cortex, 139, 1226. https://doi.org/10.1016/j.cortex.2021.02.021.CrossRefGoogle Scholar
Land, M. F. (2004). The coordination of rotations of the eyes, head and trunk in saccadic turns produced in natural situations. Experimental Brain Research, 159(2), 151160.CrossRefGoogle ScholarPubMed
Li, H. H., Barbot, A., & Carrasco, M. (2016). Saccade preparation reshapes sensory tuning. Current Biology, 26(12), 15641570.CrossRefGoogle ScholarPubMed
Ohl, S., Kuper, C., & Rolfs, M. (2017). Selective enhancement of orientation tuning before saccades. Journal of Vision, 17(13), 111.CrossRefGoogle ScholarPubMed
Ostendorf, F., Fischer, C., Finke, C., & Ploner, C. J. (2007). Perisaccadic compression correlates with saccadic peak velocity: Differential association of eye movement dynamics with perceptual mislocalization patterns. Journal of Neuroscience, 27(28), 75597563.CrossRefGoogle ScholarPubMed
Rayner, K. (2009). The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search. Quarterly Journal of Experimental Psychology, 62(8), 14571506.CrossRefGoogle Scholar
Rolfs, M. (2015). Attention in active vision: A perspective on perceptual continuity across saccades. Perception, 44(8–9), 900919.CrossRefGoogle ScholarPubMed
Rolfs, M., & Carrasco, M. (2012). Rapid simultaneous enhancement of visual sensitivity and perceived contrast during saccade preparation. Journal of Neuroscience, 32(40), 1374413752a.CrossRefGoogle ScholarPubMed
Rolfs, M., Jonikaitis, D., Deubel, H., & Cavanagh, P. (2011). Predictive remapping of attention across eye movements. Nature Neuroscience, 14(2), 252256.CrossRefGoogle ScholarPubMed
Rolfs, M., & Szinte, M. (2016). Remapping attention pointers: Linking physiology and behavior. Trends in Cognitive Sciences, 20(6), 399401.CrossRefGoogle ScholarPubMed
Rucci, M., Ahissar, E., & Burr, D. (2018). Temporal coding of visual space. Trends in Cognitive Sciences, 22(10), 883895.CrossRefGoogle ScholarPubMed
Scholes, C., McGraw, P. V., & Roach, N. W. (2021). Learning to silence saccadic suppression. Proceedings of the National Academy of Sciences, 118(6), e2012937118.CrossRefGoogle ScholarPubMed
Schweitzer, R., & Rolfs, M. (2020). Intra-saccadic motion streaks as cues to linking object locations across saccades. Journal of Vision, 20(4):17, 124.CrossRefGoogle ScholarPubMed
Schweitzer, R., & Rolfs, M. (in press). Intra-saccadic motion streaks jump-start gaze correction. bioRxiv. https://doi.org/10.1101/2020.04.30.070094.Google Scholar
Serences, J. T. (2008). Value-based modulations in human visual cortex. Neuron, 60(6), 11691181.CrossRefGoogle ScholarPubMed
van Heusden, E., Rolfs, M., Cavanagh, P., & Hogendoorn, H. (2018). Motion extrapolation for eye movements predicts perceived motion-induced position shifts. Journal of Neuroscience, 38, 82438250.CrossRefGoogle ScholarPubMed
Watson, T. L., & Krekelberg, B. (2009). The relationship between saccadic suppression and perceptual stability. Current Biology, 19(12), 10401043.CrossRefGoogle ScholarPubMed
White, A. L., Rolfs, M., & Carrasco, M. (2013). Adaptive deployment of spatial and feature-based attention before saccades. Vision Research, 85, 2635.CrossRefGoogle ScholarPubMed
Wurtz, R. H. (2008). Neuronal mechanisms of visual stability. Vision Research, 48(20), 20702089.CrossRefGoogle ScholarPubMed