R1. Self as a marketplace

No commentator seems to have a problem with my basic assumption, that (all) choice is determined by comparative valuation in a common denominator of reward, which I refer to as the competitive marketplace. There is similarly no apparent disagreement that prospective rewards are discounted hyperbolically, although some find that the resulting predictions can come just as well from other models – Hofmeyr, Nordli & Hirt, and Ross – if not better – Heyman. Proposals to modify “traditional hyperbolic discounting models” (Nordli & Hirt) include subjects' cumulative valuation by drift diffusion (in the target article, sects. 2.1, 3.1.1, but also in Huang and Harris); incorporation of or even replacement by valuation of risk (Ross, Hofmeyr); and modification – or overriding – of delay discounting by the arousal of appetite/emotion (Monterosso, Metcalfe & Jacobs, Nordli & Hirt, Strack, Deutsch, and Abraham [Strack et al.], and probably Heyman).

Huang proposes a marketplace model, or “society of controllers,” although he does not say how the controllers are determined. His controllers sound like my interests, which are learned processes named for the rewards on which they were based. He similarly proposes that they form a population, and compete for acceptance in a drift diffusion model where their expected rewards build to a threshold. Tzafestas similarly uses a bottom-up “self-organizational” model in which “higher-order” and abstract “urges can develop de novo.” Her approach is compatible with drift diffusion, but she does not mention it. I don't understand why she interprets my model as top-down – long-term interests exert power to the extent that they bid early enough or are combined enough to prevail in a competitive marketplace. Harris simply incorporates drift diffusion, which I agree is a straightforward addition to our knowledge of choice.

R2. Risk

The uncertainty of expected outcomes puts another variable in the marketplace (Ross, Hofmeyr), but this does not necessarily change the delay factor. Ross is partially right that I take time preference as a primitive, in that a monotonic effect of delay on preference is widely measured in both humans and nonhumans, and fades in human subjects only in situations that invite reflection, if not resolve. He stops short of saying that delay discounting is actually one form of risk assessment, but seems to argue that it can't be isolated experimentally. But repeated animal trials with differential delays of seconds show hyperbolic delay discounting, as do human self-reports about sure hypothetical rewards – which match their preferences for actual rewards (Johnson & Bickel, Reference Johnson and Bickel2002). Hofmeyr, at one point, rejects equating risk with delay, but again seems to argue that the dimension of delay can't be factored out, even in the laboratory, because a curved utility function blunts the ostensible size of a larger reward. But that would not produce preference reversal, which he argues would take place when the earlier of two risky rewards came closer. His narrative example introduces hyperbolic, or at least hyper-concave, discounting of risk by the back door. The same assumption is necessary for Ross' assertion that animals' apparent delay discounting “can equally be modeled as responses to uncertainty.” I am puzzled, especially because both Ross and Hofmeyr are co-authors of a recent experimental report in which risk is differentiated from delay (Harrison, Hofmeyr, Ross, & Swarthout, Reference Harrison, Hofmeyr, Ross and Swarthout2018). That delay discounting evolved as a proxy for risk assessment is a plausible just-so story, but so is the hypothesis that delay discounting is necessary to prevent information overload, and the Weber–Fechner law provided a workable mechanism off the shelf (Gibbon, Reference Gibbon1977).

Humans modify valuation of risk as well as of delay from presumably inborn elementary (primitive) calculations. We sometimes work around this endowment for hedonic purposes, for instance in savoring. In the most interesting examples, people sometimes come to value risk positively as a means of refreshing appetite, which we often can't acknowledge lest we undermine resolve (Ainslie, Reference Ainslie2003, Reference Ainslie2013a; see my response to Bieleke & Wolff).

R3. Arousal

Monterosso points out that delayed options must be evaluated from the viewpoint of various successive present moments, making resolve vulnerable to “state-based inconsistency –” the arousal of appetites or emotions. Although we remember rewards with the appetite for them factored in, as when we pack a lunch for a hike – Even satiated rats will learn a maze when hunger is turned on at the food cup by hypothalamic stimulation (Mendelsohn & Chorover, Reference Mendelsohn and Chorover1965) – current appetite is well known to make the prospect of a reward more effective. What is now being discovered is that an aroused appetite can increase the temptation for apparently unrelated rewards as well, for instance sexual arousal increasing impatience for money (Van den Bergh, Dewitte, & Warlop, Reference Van den Bergh, Dewitte and Warlop2008). That is, arousal may affect a general valuation level, a tide that raises all boats.

Some psychiatric conditions seriously change the value of delayed options in general. Deeply depressed patients sometimes say that they believe they will get better but can't imagine that happening, and so are not deterred from suicide. Conversely, someone while manic may find many imminent prospects so rewarding that no delayed contingency can compete with them. The extreme case is Monterosso's Jeckyll and Hyde, rare real-life examples of whom exist (Putnam, Reference Putnam1989). I have seen a patient show great anxiety before she switched to a dissociated alter ego, but without being able to describe the prospect. Her intertemporal bargain may have been to accede in turn-taking by two complete, incompatible sets of motives, just as habitually inhibited binge drinkers sometimes dissociate to let “the alcohol do the talking.”

The apparent specialization of brain regions in generating visceral and non-visceral motives has led many writers to advocate a bicameral decision-making process, where fast/model-free/Pavlovian/hot/impulsive systems make decisions that conflict with those of slow/model-based/instrumental/cool/reflective systems. It is easy to overstate both the internal coherence and the functional independence of these two “systems.”

Model-free or Pavlovian processes are past sequences of cues and actions that have been cached as units in memory. Although they arise irrepressibly (as demonstrated in the famous Stroop task), they also serve as macros to speed choice under familiar conditions. The brain networks that have been identified as serving model-based and model-free choice have now been found to interact intimately in real time (Dixon & Christoff, Reference Dixon and Christoff2014; Dolan & Dayan, Reference Dolan and Dayan2013). Dayan describes pragmatic collaboration of the two modes (more in Keramati et al., Reference Keramati, Smittenaar, Dolan and Dayan2016, that he cites). These “internal actions” are called upon in the same way as external actions, that is, in the marketplace of reward (an example of Bermúdez's choice among tactics). Where they do not reach a response threshold immediately, they will be part of the person's vicarious trial and error (preplay in Dayan's terms; see Schacter et al., Reference Schacter, Addis, Szpunar, Hannula and Duff2017, cited by Bulley & Schacter; Redish, Reference Redish2016) just as new plans are. Importantly, there is no reason that a person's use of model-free thinking should affect arousal of appetite or emotion per se. Model-free thinking is fast but not necessarily hot.

R4. Hot systems

Unlike model-free valuation, visceral factors intrinsically affect intertemporal bargaining. Three commentaries propose that they are the basis even of separate choice-making systems.

Hot thinking refers to the increased value of smaller, sooner (SS) rewards because of aroused appetite/emotion. Although this increase may make an SS reward more likely to overcome resolve, as Metcalfe & Jacobs point out, it does not “dissolve” resolve as they assert. They would have only the unaroused, cool “system” making – and presumably enforcing – resolutions, but the marathon runners and firefighters they cite are undoubtedly relying on resolve specifically to keep performing despite fatigue and danger. This could be said to be the main function of resolve – to push routines forward in the face of urges to abandon them. Processes that tie up cognitive capacity undoubtedly reduce the precision of intertemporal bargaining, as suggested by the Eich and Metcalfe (Reference Eich and Metcalfe2009) and Metcalfe et al. (Reference Metcalfe, Brezler, McNamara, Maletta and Vuorre2019) reports they cite, but do not disable it.

Strack et al. speak of the reflective system generating “action plans that are incompatible with the behavioral tendencies triggered by the impulsive system,” but these authors also suggest some more upstream interaction, as the reflective system “rides on top of” the impulsive. Similarly, Nordli & Hirt adopt the model of two competing brain systems, subcortical and cortical, that are separately sensitive to immediate and delayed rewards. However, although it is sometimes possible to identify input from particular brain areas in arousal of appetite/emotion, this influence does not necessarily constitute a separate valuation system. The exact shape of delay discounting remains controversial (Wulff & van den Bos, Reference Wulff and van den Bos2018): The dual discounting model of Van den Bos and McClure (cited) is contradicted by direct functional magnetic resonance imaging (fMRI) measurement of valuation, which shows no division between value-tracking of immediate and delayed rewards (Kable & Glimcher, Reference Kable and Glimcher2007). However, if a separate hot system exists, its influence must still be exerted through the common marketplace. The controversy about dualism is ultimately just over how far downstream in the choice process different rewards are compared.

It seems inevitable that interests with different payoff delays repeatedly get grouped as two different kinds, beginning with Plato's wild versus well-behaved chariot horses – sinful versus good, passionate versus reasonable, impulsive versus reflective (for instance Strack et al.), hot versus cool (for instance, Metcalfe & Jacobs), subcortical versus cortical (for instance, Nordli & Hirt). Some motivational mechanisms do serve longer term interests better than others do. However, the defining characteristic of each of these factors is its intertemporal bargaining position, and hence its strategies (in Nordli & Hirt's term). In any pairing, the longer term interest tries to forestall the shorter term interest, and the shorter term interest tries to evade this constraint. Importantly, an interest may be long term with respect to another interest, but short term with respect to still another. I've previously used the example of an ill-advised practical joke which, as an impulse, must fight off wiser choices, but must in turn act as a controller of urges so as not to spring the joke prematurely (among other examples in Ainslie, Reference Ainslie2009). Heyman points out that addictions are best seen as belonging to a middle category, impulsive with regard to a person's lifestyle aspirations, but supporting rational planning to protect the addiction. Of course for a time an addict may simply prefer “the most intense nothingness there ever was,” but having turned against it in her long-term view she may still accept failure to quit for quite a while, lest she suffer from futile attempts (see target article, sect. 3.2.1). In a more everyday example, most smokers say they want to quit, but accept the bad habit after wasting repeated efforts to do so. My point here is that such middling kinds of behavior do not dance back and forth between inhabiting hot and cool, or fast and slow, “systems.” Rather, the agent's strategy simply alternates between running with the fox and hunting with the hounds. It is between these stances that the true dichotomy lies.

Heyman also mentions my favorite example of an addictive “habit” changing instantly on the basis of different framing (Premack, Reference Premack and Hunt1970). This kind of example shows that membership in a category is the key to good versus bad “habits.” Heyman seems to think I doubt this, but what I say in the target article is that the motivation provided by belonging to a category has to be defended by seeing individual choices as test cases.

R5. Methods of impulse control

Gross and Duckworth's self-control is less specific than willpower, as they mean it to encompass all “self-initiated regulation” of impulses. Their “situation selection” and “situation modification” are upstream actions, as Kristal & Zlatev point out, whereas I have been talking about only downstream actions, the hard case: “the process of overcoming a seemingly superior, currently available SS reward,” simultaneously with the impulse (sect. 3). Even distinguishing internal self-control would not do, because this includes mental precommitting tactics such as manipulation of attention and preparation of emotion (Ainslie, Reference Ainslie2001, pp. 76–78). The latter two tactics are also part of the suppression that occurs simultaneously with an impulse, but only in that case is the motivation for them puzzling: I have argued that impulses are defined by being temporarily more valued than alternatives that pay off in the longer term, and the problem for the long-term alternatives – and for motivational theory – is how to avoid ceding control to them. In the case of suppression, Huang's proposal of a hyperdirect striatal-cortical pathway that prolongs the opportunity for suppression may be relevant.

Nordli & Hirt give examples where “it is not clear that some self-control behaviors are strictly one type or the other.” They and Bermúdez raise the question of how a person chooses a method of self-control. There are certainly choices to be made, but underneath them is a baseline of management that does not depend on deliberate action. Suppression is probably always operating to some extent. It starts with the ordinary process of maintaining intentions – of fetching objects and solving puzzles and making a sandwich – and becomes remarkable only when temptations raise obstacles to an intention without (yet) overturning it. Suppression may then grow to be an absorbing activity and show up in prefrontal cortical activity. As Huang points out, it can be directed by resolve. It does not apparently become more effective with practice (Xu et al., Reference Xu, Demos, Leahey, Hart, Trautvetter, Coward and Wing2014). However, Harris's work suggests that sequentially paying attention to particular reward attributes may be a learnable form of suppression, and even of an eyes-on-the-prize converse of suppression.

Resolve does require some self-awareness, but there is probably no adult who does not sometimes ask – or sense – the question, “if not now, when,” or “am I going to go on doing this?” without stopping to frame a resolution. However, there is considerable scope for learning to make resolve both more effective and more efficient – more effective by increasing the perceived contingency of valued outcomes on a current choice, more efficient in avoiding resolutions that are either overly restrictive or unlikely to succeed.

Maglio & Hershfield are right that intertemporal bargaining can't be a classical iterative prisoner's dilemma, because a future self can't literally retaliate against past incarnations. They suggest a dictator game, but depicting the responder as a monolithic “cumulative future self” obscures the dynamic process, imagination of which guides the present chooser. In my proof-of-concept game, a roomful of people successively chose a dollar just for oneself or ten cents added to everyone's individual total, making the cooperative, 10 cent choice adaptive if and only if the player expected similar moves by future players (Ainslie, Reference Ainslie2001, p. 93). The fact that the players were separate people rather than prospective selves separated by discounted delays did not change the logic of the contingencies they faced.

R6. Resolve versus suppression and precommitment

The self-help profession has been critical of the venerable willpower because many clients have been demoralized by fruitless attempts at “internal brute force” (Kristal & Zlatev) or Harris's “effortful moment-to-moment inhibition,” but these are not its most effective form, as I have argued. The form of willpower that has been found “least effective” (Kristal & Zlatev) is the kind so often experimented with, suppression, as in the Fujita et al., article they cite (“effortful inhibition”). Similarly, Inzlicht & Friese say that the high “trait conscientiousness” people who are “the best at meeting their goals” have done so not by willpower but by “routinization of goal-directed behaviors and cultivation of good habits,” which, I have just argued, is the outcome of successful resolve.

Lea surveys applications of resolve in real-world financial decisions, and Acquaro & Sosis in recovery from alcoholism. Khalil insists that “the term ‘willpower’ cannot be a scientific concept, that is, a concept that captures the different inner coherence of ‘resolve’ and ‘suppression’” – and, incidentally, that “resolve” denotes only higher aspirations, but he does not suggest better terms.

This is not to deny the importance of preparing in advance to face impulses (Gross & Duckworth, Kristal & Zlatev, Inzlicht & Friese) or the stressors that predispose to impulses (Dubljević & Neupert). However, impulses mostly arise from the ordinary pleasures of life – to have an extra helping, to take a break from work, to take sex play a bit further. Just starting to think about them is pleasurable. The choice to give such impulses a wider berth is not a neutral cognition but entails a loss of this pleasure. Entertainment of a risky appetite or emotion is what the Catholic church calls a venial sin, and when its avoidance isn't early enough to be accomplished by mere intention it will also require resolve. By the same token, skill at resolve lets you steer closer to danger. For instance, limited sex play is an exercise in resolve that depends on discernment of bright lines, cues that demarcate flirting and seduction, safe and unsafe sex, and teenagers' famous bases, as in “only got to first base.”

Whether boredom is a distinct emotional state, as Bieleke & Wolff propose, or just an awareness of being stuck in an unrewarding activity, is apt to be a matter of some debate. Either way, they point out that its frequent occurrence during long-term goal pursuit highlights the “exclusive focus on prediction error minimization” in reward learning theory – and utility theory generally – that fails to recognize the role of appetite in maintaining reward effectiveness. This is an important point for welfare planners. I have argued elsewhere that much seeming inefficiency in the modern world comes from indirection, the maintenance of unnecessarily challenging goals in order to refresh appetite (Ainslie, Reference Ainslie and Levy2013b).

The choice among mixed goal options as discussed by Mayer & Freund is probably the usual case, because long-term resolve will be frequently threatened by boredom, and it needs to budget subsidiary goals or harmless pastimes to stay dominant. Future prospects must compete with current comfort, and at a substantial discount. The sort of deal the authors suggest may well be effective, as long as its budgetary numbers hold up – The choice to defer a goal may be rewarding in the short term as it reduces pressure on resources, but might also be seen as a failed test case.

Practice at resolve inevitably entails failures, where you learn its limits, its routes to recovery, and how to hedge it with rationalizations and evade it by inattention to its tests. For impulses that are too strong and/or dangerous to allow failures, for instance in recovery from alcoholism, intertemporal bargaining is still called for, as in Acquaro & Sosis's “playing the tape through.” But the stake that this exercise demonstrates is Alcoholics Anonymous' “helplessness” against alcohol, that is, sobriety that is wholly vulnerable to any drink, with none of resolve's usual arbitrage permitted (see “atomic bargaining” in Ainslie, Reference Ainslie2001, pp. 113–116). Of course, advance preparation still helps – Alcoholics Anonymous' avoidance of “persons, places, and things.”

Khalil interprets how I distinguish suppression and resolve as a matter of effort. I do not present it as a defining feature, only a measure of operational cost, which may be as high in a contested resolution as in sustained suppression. Massin & Gauchot say resolve is always effortful, but they maintain that “cognitive abstraction processes always have operational costs,” even though in terms of either processing time or unpleasant experience these are often trivial. These authors also say we must stop referring to the defense of a resolution as resolve once it is successful, but this point of usage could be argued either way.

R7. Construction of the future

Besides the arousal of appetite/emotion, Monterosso points out that the other complication in figuring out the motivation for resolve is the complex basis of belief in the future. Philosopher Robert Nozick once asked why someone should care about her future selves. The answer is easy in the realm of seconds to minutes, and perhaps even hours and days – the prospect of differential reward urges us. It is probably impossible not to notice when we're about to fall into a cold lake, or even that we're facing a big exam tomorrow. More distant prospects still compete in the marketplace of reward, but their values are more subject to interpretation.

The key issue for intertemporal bargaining is what forms the stake in test cases. The most important stakes are apt to be enduring states rather than repeated LL events – freedom from the prospect of lung cancer, or a claim to good character – the contingency that Bodner and Prelec imagine in their game-theoretic analysis of self-prediction (Reference Bodner, Prelec, Brocas and Carillo2003), and that Heyman sees as the ultimately effective motivation in recovery from addiction. A history of successful resolve may itself come to form a major stake – confidence in your ability to keep resolutions. But the experience of empathy may also be worth examining as a repository of expected value.

Growing exploration of mental time travel, correlated with activity in the default network and other areas of the brain, suggest moderators of prospective value but do not promise to tell the whole story. Bulley & Schacter point out that such “episodic simulation” is probably unnecessary for “prospective cognition and deliberation.” Resolve can be enforced simply by the observation that “defection sooner implies defection later” that shows up in model-based preplay (Dayan). The overuse of resolve seen in compulsiveness is a whole other topic (Ainslie, Reference Ainslie2001, pp. 143–160), but it provides an example uncorrelated with future simulation: The rigid self-control seen in obsessive-compulsive personality disorder (OCPD) and closely correlated, autism spectrum disorder (Gadelkarim et al., Reference Gadelkarim, Shahper, Reid, Wikramanayake, Kaur, Kolli and Fineberg2019) occurs despite reduced episodic simulation (Crane et al., Reference Crane, Lind and Bowler2013), but its motivation has not been characterized.

Maglio & Hershfield explore how “the self making [future] choices remains, to the current self, a mystery.” Hershfield and collaborators have elsewhere reviewed the various ways that a current self has been proposed to experience future prospects (for instance, in Hershfield & Bartels, Reference Hershfield, Bartels, Oettingen, Sevincer and Gollwitzer2018, cited), and have noted the similarity between empathy with future selves and empathy with other people which is also found in brain imaging patterns. This is evidence for the old suggestion that we model our future selves in much the way we model other people (Hazlitt, Reference Hazlitt1805/1969; Simon, Reference Simon1995). The LL stake may be embodied in an imagined future self or a felt relationship with another person, real or imaginary, that will be spoiled by a defection. For instance, if you pray to Saint X for help in resisting an impulse, then give in, she may become less willing to help in the future (Ainslie, Reference Ainslie1975).

The further we look into the future, the less our expectations will run up against our rules for realism. Increasingly, we will be writing fiction that occasions reward endogenously, constrained by what makes fiction effective, less to be believed than believed in. But my argument for this also another topic (Ainslie, Reference Ainslie2017). I will just note here that although we construct the far future imaginatively, there is evidence that we still discount its value in hyperbolic curves, albeit not curves that span the time continuously (Ainslie, Reference Ainslie2017, p. 150).

R8. Relationship of intertemporal with interpersonal bargaining

The possibility that intertemporal bargaining evolved from the interpersonal sort is intriguing (Setman & Kelly, Veit & Spurrett). Certainly the two are related, and probably inform each other. Again, the simple observation that “defection sooner implies defection later” can occur without it, as Dayan says. Explicit pictures of a future self that is constructed of episodic memories undoubtedly add to larger, later (LL) stakes, but they are not necessary for valuing delayed rewards.

I keep my analysis within the individual person not to belittle the overwhelming importance of social mechanisms, but to make sure we keep in mind that their physical occurrence must be in individual brains. Intertemporal and interpersonal forms of bargaining can each operate without reference to the other – respectively Robinson Crusoe rationing his seed grain, and primitive tribes leaving out goods for serial, contactless barter. However, I have suggested one area where the intertemporal kind may have informed the interpersonal: Philosophers puzzle over how people who believe in strict determinism can assign moral responsibility, but we may be just broadening our natural perception of self-blame in cases of failed resolve to interpret analogous interpersonal situations (Ainslie, Reference Ainslie, Poland and Graham2011). This is the converse – and perhaps complement – of philosopher Peter Strawson's argument that individual's sense of responsibility comes from her experience in a moral community (Reference Strawson1974). Setman & Kelly propose that direction of evolution – internal bargaining copying social – which is certainly plausible.

R9. Brain imaging

I would like to think that increasing resolution in imaging of the vast core network will let us follow the conduct of resolve in real time (Bulley & Schacter). However, because resolve as I have proposed it consists of belief in a contingency (of future reward on a current choice), it is hard to picture the experimental design that would create a differential amount of brain activity directly. I made a suggestion in the target article: The best hope is probably to track resolve by a reduced use of suppression, as evidenced by a reduction in dorsolateral frontal cortex activity.

R10. Reply conclusions

Commentators brought out several important issues with willpower. First of all, the term itself looks condemned to be tarred with the implication of “internal brute force,” so it may be necessary for clarity to speak about intertemporal bargaining in so many words, or simply as resolve. Limitations of its LL stakes will be a rich field for research: State-based inconsistency can come both from the arousal of appetite or emotion and, in the extreme, from the dissociation of ego states, but probably not from dissociation of fast versus slow or hot versus cool motivational processes. Even less known is what constrains imagination of the future in creating LL stakes.

Several speculations are worth following up: The recent finding that bidding for action is not a single event but diffused among reward components suggests opportunities for temporarily less favored options to compete by instituting suppression, for instance by a hyperdirect striatal-cortical pathway that raises the decision threshold in the presence of conflict (Huang). Harris brings up a learnable route to more effective suppression: to sequentially focus attention on different reward attributes. Bieleke & Wolff raise the likelihood that reward effectiveness depends on more than prediction error minimization, which I would relate to the cultivation of appetite. Evolutionary questions include, biologically, the possible shaping effect of risk on delay discounting (Ross, Hofmeyr), and culturally, the possible learning of intertemporal bargaining from the interpersonal (Setman & Kelly, Veit & Spurrett).

This was a wide-ranging discussion of reward, which I found highly rewarding.