R1. Who computes the costs and rewards?

Spurrett notes that in order to move well, there needs to be an estimate of the relevant costs, in time, calories, and so on, and potential rewards, and then an integration into a common currency, applied to select deployment of the whole body. “Nonetheless, if there can't be a central executive able to integrate everything quickly enough to make selection and control of all skeletal movement consistently sensitive to a single value function responsive to all available information, something else must be going on.” In Vigor, we show that even in the context of an elementary movement like a saccade, there is no central executive. Rather, numerous structures, including some in the cortex, and others in the basal ganglia, express their opinions about where to move to, and how to make that movement. These opinions arrive via excitation and inhibition upon the superior colliculus, which in turn houses the neural machinery to make movements. The various opinions become potential control policies in the colliculus, with each policy translating the perceived costs and rewards into a potential goal directed movements with a specific vigor. Through a process akin to a race, one policy rises above the rest, achieves a threshold, and becomes action.

As Matsushima, Kawamori, & Ogura (Matsushima et al.) point out, the question of how the brain computes utility has been largely the focus of cortical neurophysiologists; yet fishes and birds, to name just a few species, make do quite well despite less developed cortical structures. Indeed, there is a fundamental role for the subcortical structures in decision-making. For example, in order to make a saccade, the decision-making process in mammals involves not only the attention allocation and value-based judgment of parietal and frontal lobe structures, but also the motor costs computed by the superior colliculus (Lovejoy & Krauzlis, Reference Lovejoy and Krauzlis2010). For example, a small regional deactivation of the colliculus not only makes movements to that region more difficult, but more importantly, it makes attention allocation for that region more costly. This implies that the process of decision-making is a collaboration between cortical and subcortical structures.

We might naively guess that this cortical and subcortical collaboration is proforma, because the ultimate arbiter is the cortex. Surprisingly, the opposite may be true, as illustrated by the fact that the nucleus that sets the threshold for making a saccade is not in the cortex, but in the brainstem. Omnipause neurons integrate excitatory and inhibitory inputs from the cortex, basal ganglia, and the colliculus, and then pause when the sum exceeds a threshold, thereby allowing the saccade that is encoded by the upstream neurons to take place. Thus, the fabled threshold that determines arrival of a decision to move is not in the cortex, but in the brainstem.

From an evolutionary point of view, all the basic mechanisms for decision-making, including representation of a global capture rate, risks, rewards, and effort, exist in the brainstem, thus allowing numerous species without a highly developed cerebral cortex to make good decisions that lead to fecundity. With the development of a cerebral cortex, computing utility has become more sophisticated, but not divorced from the evaluation performed in the brainstem.

R2. Competition for reward can increase vigor

Although optimal foraging provides a metabolic perspective on why movement vigor may be linked to the subjective valuation of the goal, Becchio, Pullar, & Panzeri (Becchio et al.) make the novel suggestion that our vigor can act as a signal to others regarding how we value the objects in our environment. For example, the speed of our movement toward an empty seat on the train can serve as a clear indication to others that we subjectively have placed a high value on this seat, underlining our intentions for this limited resource, which in turn may result in a reduction in their vigor. This presents a novel view on why we run toward people we love: to let them know that we love them. Thus, vigor has benefits that go beyond energetics, conveying information regarding how we value the goal at our destination.

On the day after Thanksgiving in America, people line up outside of stores early in the morning and then when the doors open, run to grab the few televisions or gaming consoles that have been advertised at exceptionally low prices. For those standing in line, the probability of acquiring this reward is low, but they are willing to expend the energy required to acquire it. As Bufacchi and Iannetti correctly point out, competition and scarcity can produce increased vigor. Indeed, as they show in their simulations, this scenario reverses the usual relationship between expected reward and vigor, but remains entirely consistent with optimal foraging theory. Thus, the theory correctly predicts that facing competition, increased vigor is justified if it increases the probability of acquiring the scare reward.

In a recent experiment (Yoon et al., Reference Yoon, Geary, Ahmed and Shadmehr2018), we tested the prediction that scarcity of reward would increase vigor. Human subjects looked at a fixation point while an image was presented to a side of the screen. To simulate scarcity, they were limited in the amount of time that they had to view the image. Thus, when they made a saccade to the image, on one block of trials they only had 0.4 s to gaze at it, whereas in other blocks they had 1.0 or 1.5 s. This modulation of harvest duration had a strong effect on both reaction time and saccade velocity: Subjects reacted sooner, and moved with greater saccade velocity toward the image that was available for the shorter period of time (Figure 3.11 of Vigor). Importantly, the expected reward in the short gaze period was smaller than in the long gaze period, but as the theory predicted, subjects moved with greater vigor toward the reward that was available for the shorter period of time. However, an important direction to explore these theories is to present a limited quantity of reward and introduce competition between agents. These richer, more ecologically relevant experiments await to be performed.

Indeed, Wright and Mlynski point out that when considering effort expenditure, the critical variables are not just the value of the incentive, but also the probability that expenditure of effort will be successful in acquiring that incentive. Is there evidence that movement vigor shows sensitivity to probability of attaining the incentive?

There is little research on this topic but there is one example that touches on this in Vigor. Seideman et al. (Reference Seideman, Stanford and Salinas2018) trained monkeys to watch a monitor in which two yellow targets appeared to the left and right of a fixation point. After a short fixation period, the fixation point disappeared, instructing the animal to go, and a reward cue appeared shortly after identifying which of the two targets was to be rewarded. Once the rewarding target was identified, the monkeys had a maximum of 450 ms following the go cue to start the saccade.

Cue processing time refers to the time from the cue onset to the saccade onset, and reflects the amount of time the monkeys had to acquire information about the identity of the rewarding target. The greater the cue processing time, the more certain the animal would be regarding its decision. Indeed, the probability of choosing the rewarding target increased with greater cue processing time (Figure 3.15 in Vigor). Interestingly, saccade vigor also increased with cue processing time suggesting that the animal was more confident that they would be rewarded. However, as cue processing time became very long, vigor decreased to baseline. This surprising result can also be explained by considering the probability of attaining reward. As cue processing time increased, the monkey was more likely to exceed the maximum time limit of 450 ms, which would remove the possibility of reward entirely. That is, as cue processing time became very long, it became less likely that the animal would be rewarded, and vigor was reduced.

Thus, in the data of Seideman et al. (Reference Seideman, Stanford and Salinas2018), we see that as cue processing time increases, the probability of incentive attainment increases, and so does saccade vigor. But, as the processing time becomes too long, the probability of acquiring the incentive drops to zero, and now saccade vigor returns to baseline. Thus, for this elementary movement, vigor investment is responsive to probability of successful attainment of the incentive.

R3. Threats can increase vigor

Bufacchi and Iannetti note that as the level of threat increases, an animal is more likely to spend energy to defend or escape that threat. Thus, avoidance of loss can lead to increased vigor. Indeed, the foraging literature documents a distinct effect of starvation threat on risk-taking behavior, with animals more likely to engage in risky behavior when they are close to starvation than when they are not (Barnard & Brown, Reference Barnard and Brown1985). A recent experiment illustrates the dual nature of serotonin in control of vigor in low- and high-threat environments.

Seo et al. (Reference Seo, Guru, Jin, Ito, Sleezer, Ho and Warden2019) optogenetically stimulated serotonin neurons during open field locomotion and noted that this caused an immediate reduction in vigor: walking speed declined. Similarly, when the mice were placed in a chamber where they had learned to associate a tone with availability of water, stimulation of the serotonin neurons reduced their approach speed toward the water spout. In contrast to these low-threat scenarios in which increase in serotonin reduced vigor, the effect of serotonin release reversed during high-threat scenarios. To produce a high-threat condition that encouraged an attempt to escape, the authors hung the animals by their tail to a horizontal bar and then measured the resulting struggle using an accelerometer. Being hung upside down naturally induced occasional bursts of rapid movements. Surprisingly, these high vigor movements accompanied high serotonin activity. In this context, stimulation of serotonin neurons increased vigor of the escape movements.

Thus, under low-threat conditions the release of serotonin reduces vigor, but under high-threat conditions the same release enhances vigor. How the possibility of threat and loss is represented by the brain, and how that representation interacts with serotonin and dopamine release to affect vigor, are fascinating topics that remain to be better explored.

R4. Building roads that connect communities of science

In the lovely analogy drawn by Matsushima et al., Roman roads were the core infrastructure that brought together various nation-states, leading to exchange of goods and ideas. In Vigor, the mathematics of optimal foraging acts as a road that connects neurophysiology of decision-making, energetics of biomechanics, and kinematics of motor control. To continue this road-building, let us summarize some of the questions raised by the reviewers, and suggest experiments that might address them:

1. 1. How does probability of reward affect vigor of movements? Decreasing probability should lower the expected value of a certain reward and correspondingly reduce vigor.

2. 2. The effort required to attain a reward can discount its value. When the brain is deliberating a decision between two options, does the effort-discounted value of each option affect the vigor of movements toward the stimulus that represents the option?

3. 3. As reward value increases, a level of difficulty (effort expenditure) that previously was too steep of a cost to pay may now become affordable. Thus, although the value of the stimulus can be discounted by its effort requirements, it is also affected by the probability that the expended effort will acquire the reward. Does vigor reflect this interplay between effort requirements and probability of incentive acquisition?

4. 4. Does vigor reflect absolute reward and effort costs, or rather is vigor a measure of reward and effort costs relative to an aspiration level?

5. 5. Do individuals infer another individual's subjective valuation of an acquisition by observing the vigor of their movements? If so, how does this affect their own valuation and hence the vigor of their own movements? In other words, does vigor serve as a social cue reflecting and driving valuation in ourselves and others?

6. 6. How does scarcity of resources influence vigor? Scarcity can lead to competition for these resources; if vigor increases the probability of reward, then scarcity should lead to greater vigor. The subjective value of a scarce resource may also be greater than the subjective value of the same resource when more abundant. In this case as well, scarcity should lead to greater vigor.

7. 7. What is the influence of threat on vigor? How do these effects interact with risky decision-making and what roles do dopamine and serotonin play in determining these effects?

Although the roads that we have built in Vigor have pitfalls, if they are to have a lasting effect, they must facilitate exchange of ideas between the fields of neuroeconomics, decision-making, and motor control. A most encouraging sign is the broad spectrum of science reflected in the reviewer comments, suggesting that this exchange has begun.