1. Introduction
To explain animal behavior, functional but also causal interpretations are necessary (Tinbergen Reference Tinbergen1963). The former attempts to determine why (for which survival or reproductive purpose) specific actions are performed, while the latter tries to determine how (by which biological and psychological mechanisms) those specific actions are performed. In this article, we examine the counterintuitive, though well-documented, evidence that individuals (at least in birds and mammals, including humans) exposed to unpredictable food supplies have higher fat reserves and/or cache more food items than individuals exposed to predictable food supplies. We show that the evolutionary origin (the why) of that phenomenon is quite well understood, but that the causal mechanisms (the how) contributing to increase fat reserves or to stimulate hoarding behavior remain largely unquestioned and therefore unknown (Pravosudov Reference Pravosudov, Stephens, Brown and Ydenberg2007). Here, we suggest a causal theory inspired from psychology and neuroscience to explain the mechanisms leading food unpredictability to enhance food seeking, a behavior that may enable animals to find more food items and hence to get fatter or to cache more items when the available amounts of food remain sufficient (e.g., Pravosudov Reference Pravosudov2003). Our causal mechanism is viewed as an adaptive consequence of the selective pressures (notably starvation and predation risks) that justify the functional interpretation put forward by behavioral ecologists. Importantly, this inquiry has the potential to also uncover processes that may underpin apparently unrelated behaviors such as drug addiction (Robinson & Berridge Reference Robinson and Berridge1993), pathological gambling (e.g., Linnet et al. Reference Linnet, Mouridsen, Peterson, Møller, Doudet and Gjedde2012), and obesity problems (Nettle et al. Reference Nettle, Andrews and Bateson2017).
The words uncertainty and unpredictability are used interchangeably and simply mean that a trial (or attempt to get food) is rewarded or nonrewarded on a random basis in a specific environment, independent of the proportion of time spent in that environment relative to another environment. An animal may experience uncertainty over repeated sessions in a Skinner box, a situation in which the exact same number of rewarded and nonrewarded trials occurs on each session. In nature, not all foraging bouts (or sessions) are likely to be similarly rewarded, especially when food is scarce. Some foraging bouts may be unsuccessful because of unfavorable meteorological conditions, whereas others are more profitable. But overall, the animal also experiences uncertainty over repeated foraging bouts in the environment in which it is used to seeking food. In all cases, uncertainty or unpredictability results in this simple – but crucial – effect: the individual's inability to predict whether the next foraging trial in a given environment will be rewarded or not. The goal of this article is to describe how organisms psychologically deal with such an absence of predictive control at the trial level, in a way that fits the functional perspective on behavior.
In behavioral ecology, hundreds of publications report that animals, like small passerines and rodents, as well as humans, accumulate more fat reserves and/or hoard more food items when their food sources are unpredictable, that is, hard to obtain and sometimes unavailable (e.g., Bauer et al. Reference Bauer, Glassman, Cyr and Romero2011; Brodin Reference Brodin2007; Cresswell Reference Cresswell2003; Cuthill et al. Reference Cuthill, Hunt, Cleary and Clark1997; Ekman & Hake Reference Ekman and Hake1990; Foster et al. Reference Foster, Solomon, Huhman and Bartness2006; Gosler Reference Gosler1996; Hurly Reference Hurly1992; Lilliendahl Reference Lilliendahl1998; Lundberg Reference Lundberg1985; MacLeod et al. Reference MacLeod, Lind, Clark and Cresswell2007; Nettle et al. Reference Nettle, Andrews and Bateson2017; Polo & Bautista Reference Polo and Bautista2006; Pravosudov Reference Pravosudov2003; Pravosudov & Grubb Reference Pravosudov and Grubb1997; Pravosudov & Lucas Reference Pravosudov and Lucas2000; Ratikainen & Wright Reference Ratikainen and Wright2013; Rogers Reference Rogers1987; Witter & Swaddle Reference Witter and Swaddle1995). Functionally, this phenomenon acts as insurance against starvation, because temporarily inaccessible food items prevent animals from meeting their daily budget requirements. For example, Hake (Reference Hake1996) found that greenfinches (Carduelis chloris) with low social status carried larger body masses than higher-ranked individuals. This occurred because dominant individuals prevented them from accessing the most predictable food sites, increasing the risk of famine among subordinate individuals. If bad weather conditions increased that risk for dominants as well, they could temporarily put on more fat than subordinates. Extra fat plays a crucial role for survival. Fatter great tits (Parus major), for example, have a better survival rate than leaner individuals in the absence of beech mast during winter (Gosler Reference Gosler1996). Hoarding behavior also provides insurance against starvation, with the advantage of external storage of food. Thus, the animal avoids the costs associated with fattening, such as a higher predation risk (Witter & Cuthill Reference Witter and Cuthill1993).
In behavioral psychology, unpredictability has not been shown to increase fat reserves, but is known to increase responding to conditioned stimuli (CSs). Specifically, a CS unreliably followed by food delivery often generates higher response rates than a CS reliably followed by food delivery (e.g., Amsel et al. Reference Amsel, MacKinnon, Rashotte and Surridge1964; Anselme et al. Reference Anselme, Robinson and Berridge2013; Boakes Reference Boakes, Davis and Hurvitz1977; Collins et al. Reference Collins, Young, Davies and Pearce1983; Gibbon et al. Reference Gibbon, Farrell, Locurto, Duncan and Terrace1980; Gottlieb Reference Gottlieb2004; Robinson et al. Reference Robinson, Anselme, Fischer and Berridge2014). Several mechanisms have been proposed to explain this effect at a causal level (Anselme Reference Anselme2015a; Hug & Amsel Reference Hug and Amsel1969; Pearce & Hall Reference Pearce and Hall1980), and there is strong evidence that food uncertainty recruits the brain reward system, in particular, the release of dopamine from the midbrain (de Lafuente & Romo Reference de Lafuente and Romo2011; Dreher et al. Reference Dreher, Kohn and Berman2006; Fiorillo et al. Reference Fiorillo, Tobler and Schultz2003; Hart et al. Reference Hart, Clark and Phillips2015; Preuschoff et al. Reference Preuschoff, Bossaerts and Quartz2006; Tan & Bullock Reference Tan and Bullock2008). Accordingly, higher dopamine levels in the brain enhance the inclination to gamble both in animals and in humans (e.g., Dodd et al. Reference Dodd, Klos, Bower, Geda, Josephs and Ahlskog2005; Johnson et al. Reference Johnson, Madden, Brewer, Pinkston and Fowler2011; Joutsa et al. Reference Joutsa, Johansson, Niemelä, Ollikainen, Hirvonen, Piepponen, Arponen, Alho, Voon, Rinne, Hietala and Kaasinen2012; Tremblay et al. Reference Tremblay, Silveira, Kaur, Hosking, Adams, Baunez and Winstanley2017). However, no functional perspective on this process has ever been discussed (e.g., Domjan Reference Domjan2005; Hollis Reference Hollis1997). In summary, two distinct research areas describe a similar phenomenon (enhanced responding to signals that food is uncertain), but one (ecology) approaches it from a functional perspective only, whereas the other (psychology) approaches it solely from a causal perspective.
Could the increase in fat reserves or in hoarding behavior observed under harsh environmental conditions (ecology) and the increase in responding to a CS in a Skinner box (psychology) be the consequences of a common underpinning mechanism? We think the answer to this question is yes. In this article, we provide a comprehensive review of the literature on the stimulating effects of food unpredictability, both in behavioral ecology and in behavioral psychology. We also discuss some neuroscientific data, because identifying brain correlates may help disentangle distinct mechanistic interpretations. On this basis, we suggest an integrative idea: Psychology and ecology describe the two faces (causal and functional) of the same coin. In other words, enhanced responding to unpredictable CSs in Pavlovian conditioning and the increased fat reserves or increased hoarding in response to unpredictable natural conditions should depend on the same causal mechanisms and therefore have the same functional purpose. From previous theoretical developments, we argue that uncertainty magnifies food-seeking motivation because, in this context, animals not only “want” to obtain rewards (Berridge & Robinson Reference Berridge and Robinson1998), but also they come to “hope” for those rewards (Anselme Reference Anselme2015a; Reference Anselme2016). The word wanting refers to the propensity to approach and physically contact a reward or its predictive CS when available. Wanting is a synonym for incentive motivation. Our central claim is that incentive hope is an extension of incentive motivation in situations in which the wanted rewards are unguaranteed on a given trial and in which uncertainty cannot be avoided. We argue that the behavioral invigoration or lengthening observed under reward uncertainty reflects a survival requirement rather than so-called preference for uncertainty. How incentive hope is related to incentive motivation is explained, as is its behavioral consequence: increasing the willingness to spend time and effort to seek uncertain rewards – in comparison with certain rewards – and their predictive cues. This mechanism provides a new causal interpretation of how foraging works, bridging the gap between animal foraging (behavioral ecology), sign-tracking behavior (behavioral psychology), and reward motivation (behavioral neuroscience). We show that this mechanism is computationally tenable, enhancing consumption and increasing fat reserves in simulated foragers seeking (pseudo)randomly distributed food items.
2. Ecology: Food unpredictability increases fat reserves and hoarding behavior
Food unpredictability causes an upregulation of body fat and/or an intensification of hoarding behavior, including humans and nonhuman mammalian species (Foster et al. Reference Foster, Solomon, Huhman and Bartness2006; Nettle et al. Reference Nettle, Andrews and Bateson2017), but most studies have focused on small passerines. For this reason, these bird species will be discussed as priority. Small passerines are characterized by a low body mass (6 g to approximately 100 g). Because of their unfavorable surface/volume ratio, these birds are subjected to a rapid loss of their internal heat when exposed to cold winter days. To maintain it, they have to eat large amounts of food, representing a gain of 7%–12% of their morning body mass (Haftorn Reference Haftorn1992). Cold may partly explain why small birds become heavier (Cuthill et al. Reference Cuthill, Maddocks, Weall and Jones2000), although they do not always put on more fat when held under cold temperatures in the laboratory (Helms Reference Helms1968; King & Farner Reference King and Farner1966; Pravosudov & Grubb Reference Pravosudov and Grubb1998). The main reason for higher fat storage in winter is that food availability is more unpredictable, increasing the risk of starvation (e.g., Gosler Reference Gosler1996; for a description of the different models, see Brodin Reference Brodin2007). This phenomenon is not specific to winter conditions; it has also been observed in subordinate individuals (e.g., Ekman & Lilliendahl Reference Ekman and Lilliendahl1993), in individuals exposed to predation risk (MacLeod et al. Reference MacLeod, Lind, Clark and Cresswell2007), and in poor foragers (Cresswell Reference Cresswell2003) – whether in the field or in captivity. Importantly, experimental manipulations that make food deprivation unpredictable, independent of temperature and dominance, can also increase fat reserves or food hoarding (e.g., Hurly Reference Hurly1992; Pravosudov & Grubb Reference Pravosudov and Grubb1997). This process of fat regulation has also been observed outside the field of behavioral ecology. In humans, words such as shortfall and adversity lead participants to consume more food items of high-energy value (Laran & Salerno Reference Laran and Salerno2013) and to express a desire to eat such items despite their absence and despite any effect on general appetite (Swaffield & Roberts Reference Swaffield and Roberts2015). Interestingly, the mere subjective feeling of lower socioeconomic status relative to others is sufficient to increase food intake and preference for high-calorie foods, irrespective of the absence of objective differences in access to financial resources (Cheon & Hong Reference Cheon and Hong2017).
Is food less abundant or of lower energy value in winter, or is it just different (seeds rather than insects) from that found by birds in summer? Often enough, the concept of food unpredictability used by behavioral ecologists is a subjective interpretation based on a human perspective instead of the result of a predictive measurement made in advance. However, we think it is justified. It is likely that the kinds of food available differ between winter and summer, especially with respect to the presence of insects. It is also possible that the seeds collected in winter contain more fat than the insects collected in summer, although the amounts of fat in larval stages can be elevated (up to 60%) in comparison with those in adults (up to 15%; Kouřimská & Adámková Reference Kouřimská and Adámková2016). But seeds are also present in summer, and certainly in greater amounts than in winter. Thus, the energy value of food may be similar in winter and summer, but the opportunities to become fatter in winter are reduced because the kinds of food available (insects and seeds) are less abundant.
Of course, fattening in a harsh, unpredictable, or unsafe environment implies that food is present in sufficient amount. For example, food insecurity in humans is associated with obesity only in high-income countries. In low-income countries, food-insecure people want to be fatter but cannot get the calories to put on weight (Nettle et al. Reference Nettle, Andrews and Bateson2017). However, “in sufficient amount” does not mean that it can easily be found (see sect. 2.2). Surviving a harsh, unpredictable, or unsafe environment implies that animals cannot reject opportunities to eat to minimize the risk of starvation – despite increased risk of predation. Indeed, fatter birds are exposed to a higher predation risk because they are slower and less agile in response to attacks (e.g., Gosler et al. Reference Gosler, Greenwood and Perrins1995; Houston et al. Reference Houston, McNamara and Hutchinson1993; King & Farner Reference King and Farner1966; Krams Reference Krams2000; Kullberg et al. Reference Kullberg, Fransson and Jakobsson1996; Lehikoinen Reference Lehikoinen1987; Lima Reference Lima1986; McNamara & Houston Reference McNamara and Houston1990). In contrast, surviving a rich, predictable, or safe environment implies that animals reject opportunities to eat to minimize the risk of predation (and other risk factors related to injury, reproduction, and so forth; see Witter & Cuthill Reference Witter and Cuthill1993) because there is no risk of starvation.
2.1. Fattening: A multifactorial process
Climate, seasonality, and body size all affect fat regulation in unpredictable environments. In large passerine birds exposed to temperate climates, such as crows (Corvus corone) and magpies (Pica pica), there is a loss of – rather than a gain in – body mass when environmental conditions are unpredictable (Acquarone et al. Reference Acquarone, Cucco, Cauli and Malacarne2002; Cucco et al. Reference Cucco, Ottonelli, Raviola and Malacarne2002). Indeed, their fat reserves are a longer-term insurance against starvation compared with smaller passerine birds; food unpredictability does not constitute an immediate danger (see Abreu & Kacelnik Reference Abreu and Kacelnik1999; Orduna & Bouzas Reference Orduna and Bouzas2004). But not all situations make larger birds indifferent to unpredictability. Corvid species living at higher latitudes, like the Siberian jay (Perisoreus infaustus), are fatter when food is unpredictable (Ratikainen & Wright Reference Ratikainen and Wright2013). This result suggests that corvid species that live and have evolved in temperate regions have no need to increase body fat under food unpredictability, because they are large and can survive without food for a while. But if similar sized birds live and have evolved in colder regions, they are likely to put on weight under food unpredictability. It is also worth noting that crows and magpies were studied in spring (Italy), when temperature and day length were increasing, whereas jays were studied early in autumn (northern Sweden), when temperature and day length were decreasing. Such seasonal and geographical differences may contribute to generate distinct patterns of fat regulation among corvid species, as observed for food hoarding in parids (Pravosudov Reference Pravosudov2006). To summarize, putting on more fat reserves under harsh environmental conditions seems to be a general rule for both avian and mammalian species. But the surface/volume ratio of the individuals and the environment in which their species has evolved together determine the degree of harshness of the environment.
2.2. The origins of fat deposition
There is good evidence that the increase in body mass under food unpredictability is essentially due to fat deposits in small passerines (Cornelius et al. Reference Cornelius, Vezina, Regimbald, Hallot, Petit, Love and Karasov2017; Gosler Reference Gosler1996). But the mechanisms underpinning fat production have remained largely unquestioned (Pravosudov Reference Pravosudov, Stephens, Brown and Ydenberg2007), essentially because functional models can fruitfully predict the dynamics of fat regulation (Sherry & Mitchell Reference Sherry, Mitchell, Stephens, Brown and Ydenberg2007). Identifying the causal mechanisms that control foraging decisions is necessary to understand how foraging works (e.g., McNamara & Houston Reference McNamara and Houston2009; Pravosudov & Smulders Reference Pravosudov and Smulders2010; Shapiro et al. Reference Shapiro, Siller and Kacelnik2008; Stephens Reference Stephens2008).
Counterintuitive, but reasonable, is the hypothesis that fat reserves increase because birds eat more when their access to food is unpredictable. Some studies reported increased consumption in harsh environments (Bauer et al. Reference Bauer, Glassman, Cyr and Romero2011; Dolnik Reference Dolnik1967; Haftorn Reference Haftorn1976; King & Farner Reference King and Farner1965; Pravosudov Reference Pravosudov2003; Pravosudov & Grubb Reference Pravosudov and Grubb1997; van Balen Reference van Balen1980). But other studies reported that fattening can occur independently of food consumption (Bednekoff & Krebs Reference Bednekoff and Krebs1995; Cornelius et al. Reference Cornelius, Vezina, Regimbald, Hallot, Petit, Love and Karasov2017; Cuthill et al. Reference Cuthill, Maddocks, Weall and Jones2000; Dall & Witter Reference Dall and Witter1998; Fokidis et al. Reference Fokidis, Burin des Roziers, Sparr, Rogowski, Sweazea and Deviche2012), suggesting that other factors, such as environment-induced changes in metabolic rates, can play a role in fat production. Small birds can decrease their metabolic rate notably by reducing body temperature and general activity (e.g., Carpenter & Hixon Reference Carpenter and Hixon1988; Dall & Witter Reference Dall and Witter1998; Pravosudov & Grubb Reference Pravosudov and Grubb1997).
The fact that fat deposition does not always result from increased food intake in an unpredictable environment does not, however, mean that animals can be lazy, seeking food unfrequently. The risk of starvation is real. Small birds have to seek food items more intensively and/or for longer durations, because locating them is a difficult task (Lovette & Holmes Reference Lovette and Holmes1995; for studies of other species, see, e.g., Daunt et al. Reference Daunt, Afanasyev, Silk and Wanless2006; Hiraldo & Donázar Reference Hiraldo and Donázar1990; Kramer & Weary Reference Kramer and Weary1991; Shettleworth et al. Reference Shettleworth, Krebs, Stephens and Gibbon1988; Tamms Reference Tamms1987). As already shown, in some species, the individuals may limit their immediate consumption and cache most items for later use (Bartness et al. Reference Bartness, Keen-Rhinehart, Dailey and Teubner2011; Cabanac & Swiergiel Reference Cabanac and Swiergiel1989; Hurly Reference Hurly1992; Lucas Reference Lucas1994; Pravosudov Reference Pravosudov2003; Shettleworth et al. Reference Shettleworth, Hampton and Westwood1995). But, whatever the strategy used, food unpredictability boosts foraging activity – with the potential consequence of increasing consumption when food is present in sufficient amounts.
3. Psychology and neuroscience: Food unpredictability promotes seeking behavior
In psychology, the invigorating effects of food unpredictability on behavior have long been noted. Here, we review the main findings obtained in some Pavlovian (autoshaping) and instrumental (free choice) procedures, and we discuss their traditional causal interpretations.
3.1. Sign-tracking, motivation, and dopamine
In Pavlovian autoshaping, an animal learns that a conditioned stimulus (CS) is predictive of the delivery of an unconditioned stimulus (UCS), such as food. Briefly, a CS is presented for a few seconds, and its termination is immediately followed by limited access to food. The animal is rewarded on every trial, whatever it actually does. What the animal does during the CS presentations is a measure of its learning and motivation to react to the CS. When a physical interaction with the CS is possible (e.g., a lever or a key, as opposed to a light or a sound), two distinct phenotypes emerge in the responses produced (Beckmann & Chow Reference Beckmann and Chow2015; Meyer et al. Reference Meyer, Cogan and Robinson2014). Some individuals come to approach and interact vigorously with the CS when it is available (rats press and nibble a lever and pigeons peck at an illuminated key) – they are called sign-trackers (Hearst & Jenkins Reference Hearst and Jenkins1974). In contrast, other individuals come to approach and interact vigorously with the food dish during the CS presentations – they are called goal-trackers (Boakes Reference Boakes, Davis and Hurvitz1977).
These intraspecific phenotype differences reflect individual brain differences, particularly with respect to the release of dopamine in the nucleus accumbens. The nucleus accumbens is a mesolimbic structure that receives part of the dopamine produced by the ventral tegmental area, which is located in the midbrain. It is homologous in mammals and birds, deriving from a common ancestor that lived more than 300 million years ago (Durstewitz et al. Reference Durstewitz, Kröner and Güntürkün1999; Reiner et al. Reference Reiner, Perkel, Bruce, Butler, Csillag, Kuenzel and Jarvis2004). Sign-trackers release more dopamine in the nucleus accumbens than do goal-trackers (Flagel et al. Reference Flagel, Watson, Robinson and Akil2007; Reference Flagel, Cameron, Pickup, Watson, Akil and Robinson2011a; Reference Flagel, Clark, Robinson, Mayo, Czuj, Willuhn, Akers, Clinton, Phillips and Akil2011b). Because dopamine in the nucleus accumbens is known to control the motivational salience of rewards and their CSs (Berridge Reference Berridge2007; Berridge & Robinson Reference Berridge and Robinson1998), sign-trackers appear to “want” rewards more than goal-trackers, causing a greater attractiveness (“wanting”) of their CSs as well (e.g., Blaiss & Janak Reference Blaiss and Janak2009; Day et al. Reference Day, Wheeler, Roitman and Carelli2006; Meyer et al. Reference Meyer, Lovic, Saunders, Yager, Flagel, Morrow and Robinson2012; Robinson & Berridge Reference Robinson and Berridge2013; Rose et al. Reference Rose, Schiffer and Güntürkün2013; Saunders & Robinson Reference Saunders and Robinson2012; Tindell et al. Reference Tindell, Smith, Berridge and Aldridge2009). The differences between sign- and goal-trackers are basically unrelated to the question of uncertainty processing, but we will show that they are important in understanding the mechanism underlying an animal's responses to reward uncertainty (e.g., Anselme et al. Reference Anselme, Robinson and Berridge2013; Gottlieb Reference Gottlieb2005).
3.2. Sign-tracking and uncertainty
Since the work of Amsel and Roussel (Reference Amsel and Roussel1952), psychologists have found that Pavlovian responses are magnified when cues are unreliable predictors of reward. In Pavlovian autoshaping, sign-tracking often comes to reach a higher asymptotic level when a CS is unsystematically followed by a UCS than when a CS is always followed by a UCS (Amsel et al. Reference Amsel, MacKinnon, Rashotte and Surridge1964; Anselme et al. Reference Anselme, Robinson and Berridge2013; Boakes Reference Boakes, Davis and Hurvitz1977; Collins et al. Reference Collins, Young, Davies and Pearce1983; Crawford et al. Reference Crawford, Steirn and Pavlik1985; Gibbon et al. Reference Gibbon, Farrell, Locurto, Duncan and Terrace1980; Gottlieb Reference Gottlieb2004; Reference Gottlieb2006; Robinson et al. Reference Robinson, Anselme, Fischer and Berridge2014; 2015; Swan & Pearce Reference Swan and Pearce1987; Torres et al. Reference Torres, Glueck, Conrad, Moron and Papini2016). The effect is easy to replicate, although some studies failed to obtain it (e.g., Papini & Overmier Reference Papini and Overmier1984; Reference Papini and Overmier1985; Rescorla Reference Rescorla1999). These results suggest that animals unconsciously learn an averaged pattern from their past experience to represent the predictive accuracy of the CS, as an artificial neural network could do (Bechtel & Abrahamsen Reference Bechtel and Abrahamsen1991). At the session level, a 50% chance of reward appears to be a probability value without any particularity, and nothing of interest could be predicted with respect to response rates. But at the trial level, a 50% chance of reward means that the individual cannot expect rewards more than non-rewards; uncertainty is maximal with respect to a specific CS. Animals seem to use this information to control the strength of their sign-tracking responses on the next trial. But which psychological process can account for this effect? At first sight, the elevated asymptotic performance under uncertainty is difficult to explain in terms of incentive salience: Why would animals be more motivated to approach and interact with a CS that fails to predict food on each trial? Several causal mechanisms have been proposed to account for this effect. These explanations invoke mechanisms of frustration, attention, or motivation. We now dwell on these hypotheses and argue that the motivational perspective is the best option to fully account for sign-tracking responses under partial reinforcement.
3.2.1. Frustration
An influential view is frustration theory (Amsel Reference Amsel1958). Frustration is assumed to develop when there is a violation of an expected reward and to produce some behavioral reactions to the absence of reward (Amsel Reference Amsel1958; Reference Amsel1992). Initially directed to the food dish (unconditioned frustration), frustration can be learned and directed to the CS (conditioned frustration). Here, a frustration drive develops and magnifies the dominant responses, for example, sign-tracking. Over training, conditioned frustration can then be gradually counterconditioned by the occasional delivery of reward, a process known to convert avoidance into approach behavior. In this section, we are not criticizing frustration theory as such, but only its prediction that enhanced responding under uncertainty would result from the frustration drive or from the counterconditioning of conditioned frustration. First, we think that reward uncertainty does not make room for a frustration drive to develop. Experiencing frustration involves the violation of a strong expectation, as in extinction and successive negative contrast, in which the amount or concentration of an expected reward is suddenly decreased. However, there is no strong expectation under a 50% probability of reinforcement, because, as noted earlier, reward cannot be expected more than non-reward on each trial (Anselme Reference Anselme2015a). Second, this theory does not clearly tell us how counterconditioning could increase partially reinforced responses at a higher level than the continuously reinforced responses. Counterconditioning can eliminate (“counter”) the frustration associated with the anticipation of non-rewards. But counterconditioning does not provide “extra fuel” to boost performance when avoidance is reduced to zero, that is, when partially reinforced individuals come to respond like continuously reinforced individuals. Torres et al. (Reference Torres, Glueck, Conrad, Moron and Papini2016) found that massive lesions of the dorsomedial striatum eliminate the higher asymptotic performance under uncertainty, but have no effect on performance in extinction or after successive negative contrast. This result suggests that reward uncertainty is processed differently from reward omission and reward devaluation, which are more likely to result in frustration experience.
3.2.2. Attention
Another important theory of the behavioral effects of reward uncertainty is that unreliable CSs attract attention more than reliable CSs (Pearce & Hall Reference Pearce and Hall1980; Pearce et al. Reference Pearce, Kaye, Hall, Commons, Herrnstein and Wagner1982). An orienting attentional response occurs when the actual outcome does not fit the expected outcome, in order to favor learning. This view has been supported by a number of findings (e.g., Collins et al. Reference Collins, Young, Davies and Pearce1983; Kaye & Pearce Reference Kaye and Pearce1984), and we are not trying to call it into question. But we think that the attentional response is a correlate rather than the cause of enhanced behavioral responses. In their model, Pearce et al. (Reference Pearce, Kaye, Hall, Commons, Herrnstein and Wagner1982) suggested that sign-tracking is controlled not only by the associative strength of a CS with a UCS, which is lower under uncertainty, but also by an orienting attentional response, supposed to be higher under uncertainty. The combination of the two processes is hypothesized to increase performance with unreliable CSs. A weak point of the theory is that it does not tell us how associative strength and the orienting response interact to control sign-tracking, so that whether the orienting response is sufficient to compensate for the decreased associative strength resulting from reward uncertainty cannot be predicted (Collins & Pearce Reference Collins and Pearce1985). Another problem is the difficulty in understanding how (and why) attention itself should influence behavior. Although attention has a focus, it is a nonspecific process in the sense that attention can be allocated to any stimulus – whether it is appetitive or aversive. Therefore, how could the same nonspecific attentional process explain, for example, that animals approach and interact with a CS+ (rewarded), while leading them away from a CS– (nonrewarded)? Uncertainty is likely to recruit attention more intensely than certainty, but we think that attentional arousal in this context can only be the consequence of a more basic process capable of explaining both the directedness and the strength of behavioral responses.
3.2.3. Evidence for a motivational process
Some findings suggest that the “basic process” in question is related to incentive motivation. For example, rats trained under reward uncertainty will sign-track on a lever CS located at a longer distance from the food dish than rats trained under reward certainty, suggesting that the CS has acquired a higher motivational salience (Robinson et al. Reference Robinson, Anselme, Fischer and Berridge2014). The uncertainty effect on asymptotic performance is maintained, even after reducing the uncertainty level to which the rats were initially exposed, suggesting that uncertainty sensitizes the brain reward circuit in a similar way to dopaminergic drugs (Robinson et al. Reference Robinson, Anselme, Fischer and Berridge2014). Such a long-lasting effect of uncertainty training was also observed after changing the initially trained CS or the initially trained reward contingency from uncertainty to certainty (Gottlieb Reference Gottlieb2006). Accordingly, reward uncertainty generates a larger number of sign-trackers and stronger sign-tracking responses than reward certainty, and elevates sign-tracking in a similar fashion to amphetamine, a dopamine agonist-like drug (Robinson et al. 2015). Also, uncertainty and dopamine have facilitating effects on each other in behavioral tasks (Singer et al. Reference Singer, Scott-Railton and Vezina2012; Zack et al. Reference Zack, Featherstone, Mathewson and Fletcher2014). In addition to these behavioral facts, it is worth noting that many neurophysiological studies reveal that mesolimbic dopamine release is higher when the unreliability of a CS is maximal (de Lafuente & Romo Reference de Lafuente and Romo2011; Dreher et al. Reference Dreher, Kohn and Berman2006; Fiorillo et al. Reference Fiorillo, Tobler and Schultz2003; Hart et al. Reference Hart, Clark and Phillips2015; Preuschoff et al. Reference Preuschoff, Bossaerts and Quartz2006; Tan & Bullock Reference Tan and Bullock2008). The fact that lesions of the dorsomedial striatum – with its abundance of dopamine receptors – specifically cancel the adjustment to reward uncertainty (Torres et al. Reference Torres, Glueck, Conrad, Moron and Papini2016) may suggest that a motivational process related to incentive salience controls uncertainty processing.
Of course, it is not satisfactory to say that reward uncertainty enhances the motivation to sign-track without providing an original mechanism that is compatible with – though, partly different from – the incentive salience hypothesis. This mechanism is discussed in section 4. Importantly, for instance, we are not trying to suggest that reward uncertainty is attractive in (or sought for) itself. A number of studies indicate that animals may prefer a probabilistic option to a certain option when given a free choice (Belke & Spetch Reference Belke and Spetch1994; Dunn & Spetch Reference Dunn and Spetch1990; Gipson et al. Reference Gipson, Alessandri, Miller and Zentall2009; Laude et al. Reference Laude, Stagner and Zentall2014; Mazur Reference Mazur1991; Pattison et al. Reference Pattison, Laude and Zentall2013; Spetch et al. Reference Spetch, Belke, Barnet, Dunn and Pierce1990; Stagner & Zentall Reference Stagner and Zentall2010; Vasconcelos et al. Reference Vasconcelos, Monteiro and Kacelnik2015). But in most of these studies, the animals choose the probabilistic option only if it is associated with reliable CSs in the terminal link (e.g., if the white CS turns red, then 100% chance of reward; if the white CS turns green, then 0% chance of reward) and the surer option with unreliable CSs in the terminal link (e.g., if the white CS turns yellow or blue, then 75% chance of reward; for an excellent review, see McDevitt et al. Reference McDevitt, Dunn, Spetch and Ludvig2016). Preference is reversed when reward contingencies are reversed, and indifference is shown when the two options contain reliable CSs (Chow et al. Reference Chow, Smith, Wilson, Zentall and Beckmann2017; Smith & Zentall Reference Smith and Zentall2016). In other words, animals are not attracted by uncertainty or even by the amount of food that can be obtained; they track the reliability of CSs – an ability that is crucial for survival in the wild. Thus, we draw the conclusion that uncertainty-induced motivation in autoshaping cannot be equated with a preference for uncertainty.
Incentive salience is sufficient to explain choice behavior in probabilistic schedules, because animals are simply attracted by design elements with incentive salience – the reliable CSs. However, this view is unlikely to explain “contrafreeloading,” the well-documented fact that animals may prefer earned over free food (Inglis et al. Reference Inglis, Forkman and Lazarus1997). For example, gerbils spend more time foraging and consume more items from a bowl containing 200 seeds mixed with sand than from a bowl containing 1,000 seeds without sand (Forkman Reference Forkman1991; Reference Forkman1993). Here, the earned-food option is not associated with any attractive elements that could motivate preference. In addition, contrafreeloading is more frequent when food deprivation is low, suggesting that more than incentive salience is required to account for it. Current evidence supports the information primacy model (Inglis Reference Inglis, Archer and Burke1983; Woodworth Reference Woodworth1958), which posits that contrafreeloading results from a “need to know” aimed to reduce uncertainty (Inglis et al. Reference Inglis, Langton, Forkman and Lazarus2001). Indeed, contrafreeloading is observed only when the unprofitable/earned-food source is hidden (in sand, under lids, etc.) or changed in location over the trials; if it is visible or unchanged, the more profitable/free food source is preferred (Bean et al. Reference Bean, Mason and Bateson1999; Forkman Reference Forkman1996; Havelka Reference Havelka1956). Contrafreeloading experiments indicate that animals do not like uncertainty, as also suggested by the incentive hope hypothesis. Here, they choose to work harder in the uncertain option not because it is associated with attractive elements, as in the case of probabilistic choice schedules, but because this is the natural way of countering the adverse effects of uncertainty – similarly to rats trained under partial reinforcement in a Skinner box. Incentive hope for exploitable information might be an appropriate expression to characterize this “need to know.”
3.3. Sign-tracking as a predictor of exploratory activity
Animals use various CSs in their environment to predict the presence of food, for example, the holes of earthworms, the odor of fruits, and the sounds of flying insects (e.g., Feenders & Smulders Reference Feenders and Smulders2011; Heppner Reference Heppner1965; Wenzel Reference Wenzel1968). Often enough in nature, however, CSs are only imperfect predictors of food because they may persist long after a potential prey is gone (e.g., earthworms’ holes) or because they are associated with an unpalatable or a dangerous prey (e.g., the sound of a flying hornet), causing repeated failures in the attempts to obtain prey.
Can the propensity to sign-track depend on the reliability of CSs? It is reasonable to argue that the more unpredictable the food items, the more sign-tracking behavior makes sense. Indeed, an animal should not reject opportunities to eat in this unfavorable context and should track all potential food sources. This is in accordance with the evidence that more rats become sign-trackers and provide stronger sign-tracking responses under uncertainty (Robinson et al. 2015). Because the motivational salience of a CS is computed independent of its predictive value (e.g., Flagel et al. Reference Flagel, Watson, Robinson and Akil2007; Robinson & Berridge Reference Robinson and Berridge2013; Robinson & Flagel Reference Robinson and Flagel2009), animals can potentially seek CSs more quickly and/or for longer periods under uncertainty, even if they learned that those CSs are not reliable. If enhanced sign-tracking responses under uncertainty reflect such natural conditions, a positive correlation between the propensity to sign-track and exploratory activity should therefore be expected. Indeed, there is a positive correlation between sign-tracking behavior and novelty place preference (Beckmann et al. Reference Beckmann, Marusish, Gipson and Bardo2011), as well as with the propensity to travel in an open field (Dickson et al. Reference Dickson, McNaughton, Hou, Anderson, Long and Chesler2015). Flagel et al. (Reference Flagel, Robinson, Clark, Clinton, Watson, Seeman, Phillips and Akil2010) reported a strong sign-tracking propensity in rats selectively bred to be high responders (bHRs) to novelty compared with rats selectively bred to be low responders (bLRs). Interestingly, bHRs had a greater density of dopamine D2 receptors in the striatum and showed more spontaneous dopamine release in the core region of the nucleus accumbens than bLRs. In short, sign-tracking as an index of incentive motivation might also be a reliable index of exploratory activity.
We are not aware of any studies analyzing autoshaping-like situations in the wild, although we have provided some evidence that both might be related (see also Suzuki Reference Suzuki1986). But autoshaping is in line with a general principle of behavioral ecology that, in nature, food items are encountered sequentially rather than simultaneously (Shapiro et al. Reference Shapiro, Siller and Kacelnik2008; Stephens Reference Stephens2008; Stephens & Krebs Reference Stephens and Krebs1986). In serial autoshaping, the trials (CS and food delivery) are indeed presented one after the other, interspersed by an intertrial interval. Autoshaping is certainly an oversimplified procedure in many respects (e.g., Stephens & Anderson Reference Stephens and Anderson2001; Stephens et al. Reference Stephens, Kerr and Fernandez-Juricic2004), but Shapiro et al. (Reference Shapiro, Siller and Kacelnik2008) found that the best predictive model of foraging performance is achieved by what they call the sequential choice model (see also Freidin et al. Reference Freidin, Aw and Kacelnik2009; Vasconcelos et al. Reference Vasconcelos, Monteiro, Aw and Kacelnik2010). Thus, it is predicted that autoshaping measures a real phenomenon and gives us some relevant pictures of animal foraging in the wild.
3.4. The crucial importance of delays for food
Despite the importance of CSs in driving animal behavior, the delays for food are also crucial in determining the ability to survive. When two rewarded options are tested separately (no-choice trials), the option that generates a lower response latency is often that selected preferentially when both are presented simultaneously (choice trials; Shapiro et al. Reference Shapiro, Siller and Kacelnik2008). This means that an animal will attempt to reduce the delay to obtain a “wanted” food item. Similarly, a reward delivered after a short delay is more attractive than the same reward delivered after a longer delay (e.g., Cardinal Reference Cardinal2006; Estle et al. Reference Estle, Green, Myerson and Holt2006; Mazur Reference Mazur, Commons, Mazur, Nevin and Rachlin1987). According to Mazur (Reference Mazur, Commons, Mazur, Nevin and Rachlin1987), a reward loses attractiveness as a function of the time elapsed between a response and reward delivery (temporal discounting), and this phenomenon can be represented by the hyperbolic equation v = a/(1 + kd), where v is the subjective value of the delayed reward, a is the amount of that reward, k is a slope adjustment factor, and d is the delay value (Fig. 1). The activity of dopamine neurons reflects temporal discounting, with shorter delays being associated with stronger dopamine release (Day et al. Reference Day, Jones, Wigthtman and Carelli2010; Hariri et al. Reference Hariri, Brown, Williamson, Flory, de Wit and Manuck2006; Kobayashi & Schultz Reference Kobayashi and Schultz2008; Roesch et al. Reference Roesch, Calu and Schoenbaum2007).
Figure 1. The subjective value V of a delayed reward decreases in a hyperbolic fashion as the delay (or time t) before receiving that reward increases. As a result, a variable-delay schedule (reward after t – n or t + n) is preferred over a constant-delay schedule (reward after t) equal to its mean. Here, V(t) is smaller than the mean subjective value between V(t + n) and V(t – n) – a property called Jensen's inequality – because of the high attractiveness of immediate or rapid rewards (received at t – n) in comparison with more delayed rewards (received at t + n).
When animals are given a choice between a constant-delay and a variable-delay option, they prefer the variable delay (Kacelnik & Bateson Reference Kacelnik and Bateson1996), and even an unpredictably variable delay to a predictably variable delay (Bateson & Kacelnik Reference Bateson and Kacelnik1997). The reason for that preference is well accounted for by temporal discounting: Variability occasionally allows a quicker delivery of food (Fig. 1). A preference for variability over constancy is also often observed with respect to the ratio of responses to provision of food, an option that potentially combines smaller effort and shorter delay. Some studies even report a preference for variable ratios when variability is associated with lower gains or greater effort than constancy (e.g., Ahearn et al. Reference Ahearn, Hineline and David1992; Field et al. Reference Field, Tonneau, Ahearn and Hineline1996; Johnson et al. Reference Johnson, Madden, Brewer, Pinkston and Fowler2011), suggesting that the assessment of delays is of primary importance during foraging. Also, there is more chance for a random-ratio schedule to be preferred to a fixed-ratio schedule if the mean number of responses required for food is elevated (Madden et al. Reference Madden, Dake, Mauel and Rowe2005), because in that case, the items quickly received make a clear-cut difference with the fixed option. Dopaminergic drugs may increase the preference for variable-ratio over fixed-ratio schedules (Johnson et al. Reference Johnson, Madden, Brewer, Pinkston and Fowler2011; see also Anselme et al. Reference Anselme, Edes, Tabrik and Güntürkün2018), probably because these drugs increase the motivational salience of immediate rewards while having almost no effect on delayed rewards.
Overall, the attractiveness of shorter delays for food is compatible with the incentive salience hypothesis that animals should “want” a quick food item more than a delayed one. In fact, animals always prefer quicker food, even if food delivery is not associated with variability (e.g., Lea Reference Lea1979). This indicates that variability is not sought for itself. Preference for variable delays is comparable to that of probabilistic outcomes, discussed earlier (sect. 3.2.3), although the event that controls preference is different: In both cases, variability is unimportant as such; animals are just tracking properties that favor their survival (short delay or CS reliability). Functionally, organisms prefer rapid, easy rewards for at least two reasons: (1) They provide energy in case an emergency or a good opportunity occurs, and (2) delayed, costly rewards are less likely to be obtained because of intraspecific and interspecific competition. So, exploiting the immediately available resources is often an optimal strategy, even if those resources are in small amounts. Delays are perhaps even more important than CSs because long delays (scarce food) associated with reliable CSs are likely to imperil survival to a larger extent than short delays (abundant food) associated with unreliable CSs.
4. Uncertainty and food-seeking motivation: Causal and functional implications
We saw that psychologists developed mechanistic (causal) theories that account for behavioral invigoration under reward uncertainty. However, some of those theories have a series of shortcomings, suggesting that they are at best incomplete. In this section, we propose a mechanistic theory, initially restricted to autoshaping situations, to explain the psychological underpinning of this behavioral effect (Anselme Reference Anselme2015a; Reference Anselme2016). We show how it can be extended to natural environmental conditions, with the functional effect of regulating fat reserves and/or hoarding behavior in foraging individuals. Briefly, we argue that increased foraging activity under uncertainty reflects a stronger motivation to seek food. It is assumed that this psychological mechanism (called incentive hope) was shaped by natural selection as an insurance against starvation.
4.1. Chronic stress and its motivational correlate
Unpredictable food might be one of the numerous stressors animals encounter in their environment (Gosler Reference Gosler1996; Jenni-Eiermann et al. Reference Jenni-Eiermann, Glaus, Gruebler, Schwabl and Jenni2008; Marasco et al. Reference Marasco, Boner, Heidinger, Griffiths and Monaghan2015; Pravosudov et al. Reference Pravosudov, Kitaysky, Wingfield and Clayton2001; Strochlic & Romero Reference Strochlic and Romero2008). In this section, we briefly discuss the physiology of stress and consider its impact on motivated behavior. Chronic stress activates the hypothalamus-pituitary-adrenocortical (HPA) axis, leading to the increased production of glucocorticoid hormones such as corticosterone and cortisol (Cabib & Puglisi-Allegra Reference Cabib and Puglisi-Allegra2012). Although some studies failed to show an increase in plasma corticosterone levels in small birds exposed to unpredictable food access (Bauer et al. Reference Bauer, Glassman, Cyr and Romero2011; Partecke et al. Reference Partecke, Schwabl and Gwinner2006), others established a positive correlation between these two parameters (Jenni-Eiermann et al. Reference Jenni-Eiermann, Glaus, Gruebler, Schwabl and Jenni2008; Marasco et al. Reference Marasco, Boner, Heidinger, Griffiths and Monaghan2015; Pravosudov Reference Pravosudov2003; Reneerkens et al. Reference Reneerkens, Piersma and Ramenofsky2002), but changes in body mass were not systematically observed (e.g., Marasco et al. Reference Marasco, Boner, Heidinger, Griffiths and Monaghan2015; Reneerkens et al. Reference Reneerkens, Piersma and Ramenofsky2002). Specifically, glucocorticoids contribute to boost foraging activity. For example, corticosterone treatments increase locomotion in a novel – but not in a familiar – environment in rats (Sandi et al. Reference Sandi, Venero and Gauza1996) and extend home ranges in territorial white-crowned sparrows (Breuner Reference Breuner1998). Higher levels of corticosterone speed up exploration in zebra finches (Martins et al. Reference Martins, Roberts, Giblin, Huxham and Evans2007) and facilitate food-caching behavior and food consumption in mountain chickadees (Pravosudov Reference Pravosudov2003).
In other words, moderate elevation in baseline levels of glucocorticoids relative to food uncertainty might enhance exploratory activity and feeding (Reneerkens et al. Reference Reneerkens, Piersma and Ramenofsky2002). This phenomenon can be understood as stemming from the well-documented fact that glucocorticoids boost dopamine release from the ventral tegmental area (Barrot et al. Reference Barrot, Marinelli, Abrous, Rougé-Pont, Le Moal and Piazza2000; Piazza et al. Reference Piazza, RougePont, Deroche, Maccari, Simon and LeMoal1996; Rougé-Pont et al. Reference Rougé-Pont, Deroche, Le Moal and Piazza1998), increasing dopamine levels mainly in the shell region of the nucleus accumbens (Cabib & Puglisi-Allegra Reference Cabib and Puglisi-Allegra2012). As already discussed, mesolimbic dopamine enhances an individual's incentive motivation (or “wanting”) to approach and physically contact rewards and their associated CSs (Berridge & Robinson Reference Berridge and Robinson1998; Flagel et al. Reference Flagel, Watson, Robinson and Akil2007; Robinson & Berridge Reference Robinson and Berridge2013; Tindell et al. Reference Tindell, Smith, Berridge and Aldridge2009). Given that sign-tracker rats have higher corticosterone levels than individuals for which the CS was unpaired with food delivery (Tomie et al. Reference Tomie, Silberman, Williams and Pohorecky2002; Reference Tomie, Tirado, Yu and Pohorecky2004), it is conceivable that, rather than glucocorticoids, dopamine pharmacologically controls exploratory activity. Indeed, Piazza et al. (Reference Piazza, RougePont, Deroche, Maccari, Simon and LeMoal1996) found that peripheral administration of corticosterone increased extracellular concentrations of dopamine and locomotion in rats. These effects were more pronounced during a rewarding activity such as eating and drinking than in the absence of rewards. However, corticosterone-induced locomotion was suppressed following massive lesions of the dopamine neurons in the nucleus accumbens by means of the neurotoxin 6-hydroxydopamine. Accordingly, a number of studies indicate that there is a strong interaction between stress hormones and rewards, including food, sex, and drugs of abuse (e.g., Bronson & Desjardins Reference Bronson and Desjardins1982; Fuller & Snody Reference Fuller and Snody1981; Honma et al. Reference Honma, Honma and Hiroshige1984; Krieger Reference Krieger1974; Oswald et al. Reference Oswald, Wong, McCaul, Zhou, Kuwabara, Choi, Brasic and Wand2005; Peciña et al. Reference Peciña, Schulkin and Berridge2006). It should also be noted that high-anxiety rats sign-track more under reward uncertainty than their low-anxiety counterparts, and more than high-anxiety rats trained under reward certainty, although dopamine and corticosterone levels have not been measured (Hellberg et al. Reference Hellberg, Levit and Robinson2018). Based upon this analysis, we suggest that the motivational consequences of glucocorticoids on dopamine release (rather than stress itself) boost foraging performance. In the next section, we characterize the type of motivation required to potentially increase food consumption and/or food hoarding in unpredictable environments.
4.2. The incentive hope hypothesis
The concept of incentive hope was originally used to explain behavioral invigoration under uncertainty in Pavlovian autoshaping (Anselme Reference Anselme2015a; Reference Anselme2016). But we think that this autoshaping-based effect is only part of the whole story, and therefore the use of this concept should be extended to animal foraging in general. After briefly describing the concept of incentive hope, we show that the uncertainty of food availability in natural environmental conditions may recruit the same brain mechanisms as the probabilistic uncertainty of food in autoshaping.
4.2.1. The concept of incentive hope
In its canonical form, the incentive salience hypothesis does not make any prediction about the effects of reward uncertainty on behavioral performance. Also, if reward uncertainty just increased incentive salience, it should be wrongly predicted that uncertainty is sought for itself and preferred as such over certainty in a concurrent reinforcement task. Thus, a new concept, encompassing (but not reducible to) that of incentive salience, is required to account for the motivational properties of reward uncertainty. Elsewhere, one of us suggested that animals exposed to uncertainty are not only attracted by rewards (as they are under certainty) but, in a sense, also “hope” for their delivery (Anselme Reference Anselme2015a; Reference Anselme2016). Initially, this concept was purely descriptive: Having a motivation for an unguaranteed reward is exactly what hope means. But it is also explanatory and predictive, as shown further. The rationale behind that concept is similar to that behind the concept of “wanting” proposed by Kent Berridge and Terry Robinson in the 1990s. “Wanting” is the core motivational process that controls our conscious desires, except that its occurrence does not involve any knowledge or any subjective feeling (Anselme & Robinson Reference Anselme and Robinson2016; Berridge Reference Berridge and Kahneman1999; Reference Berridge2007). It denotes what motivation-without-cognition/consciousness is. Accordingly, “wanting” and conscious desires have the same behavioral properties: They lead individuals to approach and contact rewards, as well as their predictive cues. Relying on “wanting,” incentive hope is related to conscious hopes in exactly the same way: Organisms exposed to uncertainty behave as if they explicitly hoped for a reward, but for that, they do not have to experience hope in its full psychological (human) sense. Glucocorticoid-induced dopamine release is assumed to be the ground on which incentive hope adds its motivational effects to those of “wanting,” causing a faster approach and a more vigorous interaction with the CSs and the rewards, when available. In autoshaping, for example, partial reinforcement increases responding to a CS because animals hope that the trial will be rewarded – in other words, that the CS will be reliable. Incentive hope basically means that an animal is in a state of motivational excitement for possible good news (rewards) when bad news (non-rewards) is likely.
The incentive hope hypothesis focuses on rewards to come, whereas frustration theory focuses on lost rewards. Given that frustration is likely to be a cause of stress, introducing glucocorticoids (a neurobiological marker of stress) as a ground on which incentive hope can develop may appear surprising (see sect. 4.1). However, the two phenomena seem to have distinct effects on dopamine release. Frustration generates some avoidance of the CS, and this should induce a decrease in mesolimbic dopamine levels, as described in rats subjected to a successive negative contrast procedure (Genn et al. Reference Genn, Ahn and Phillips2004; see also Leszczuk & Flaherty Reference Leszczuk and Flaherty2000). This effect occurs when a strong expectation of reward is violated. But we suggested that non-rewards are processed differently under uncertainty, because there is no strong expectation of rewards on a given trial (see sect. 3.2.1). Many situations in which we experience stress are related to the uncertainty of an outcome (taking an exam, having an appointment, talking in public, having a medical examination, etc.). In those situations, uncertainty is not a source of frustration; instead, we hope for passing the exam, being at the appointed time, giving an interesting talk, having no medical problem, and so forth. Thus, dopamine levels are assumed to increase (rather than decrease) as a motivational consequence of uncertainty (e.g., Fiorillo et al. Reference Fiorillo, Tobler and Schultz2003; Hart et al. Reference Hart, Clark and Phillips2015). Accordingly, food uncertainty increases glucocorticoid levels (Coover et al. Reference Coover, Murison, Sundberg, Jellestad and Ursin1984), while stimulating approach behavior (e.g., Anselme et al. Reference Anselme, Robinson and Berridge2013; Gottlieb Reference Gottlieb2004).
Does incentive hope require learning? Detecting some uncertainty in reward distribution requires learning something about reward contingency, just as the attribution of incentive salience to a CS is only possible only if the animal learned to associate its presentation with food delivery (e.g., Fiorillo et al. Reference Fiorillo, Tobler and Schultz2003; Sunsay & Rebec Reference Sunsay and Rebec2008). But what has been learned about an event does not control the strength of approach or even whether the event will be approached; only incentive salience modulates that behavior (Berridge Reference Berridge2012; Robinson & Berridge Reference Robinson and Berridge2013; Tindell et al. Reference Tindell, Smith, Berridge and Aldridge2009; Zhang et al. Reference Zhang, Berridge, Tindell, Smith and Aldridge2009). The same is true of incentive hope as a motivational process: Uncertainty-induced dopamine release – a non-learning process in itself – is assumed to be the ground on which incentive hope can develop. Reward uncertainty must somehow be learned, but only the motivational effects of uncertainty (incentive hope) are assumed to modulate foraging activity. In short, we recognize the primary importance of learning in an organism's ability to develop incentive hope, but understanding how incentive hope relates to behavioral performance does not require any direct reference to learning mechanisms.
As discussed earlier, it is important to realize that incentive hope does not motivate animals to “want” unpredictable situations; incentive hope only motivates animals to seek food more intensively and/or for longer (to work harder) when unpredictability is unavoidable, such as in autoshaping and in real environmental conditions. If unpredictability is avoidable, as in a free-choice task in which an animal must choose between an unreliably signaled 50% and a reliably signaled 100% chance of reward, our hypothesis predicts that the animal will prefer the predictable option – why hope for something unguaranteed if that something can be obtained for sure? (At best, the 50% and 100% options will generate a similar number of responses because of counterconditioning in the 50% option; see Anselme Reference Anselme2016.) As already explained, current evidence supports this view: When variable-delay and probabilistic outcomes are chosen in free-choice tasks, they are chosen for reasons unrelated to their lack of constancy (such as reliable CSs and short delays). Because of this, the concept of incentive hope is fully compatible with optimal foraging theory. Incentive hope is an adaptive process – shaped by natural selection – allowing animals and humans to deal with unavoidable uncertain outcomes, reducing their negative effects on survival through invigoration or lengthening of seeking behavior. Of course, in autoshaping, responding more to an unreliable CS does not allow the animal to collect more rewards. The animal is simply exploiting a behavioral strategy put in place by evolution, urged by the presence of reward uncertainty.
It could be argued that the concepts of incentive hope and prediction error (Schultz Reference Schultz1998) make identical predictions with respect to dopamine release. This is true, but error theory does not predict an increase in conditioned responding under uncertainty, because enhanced dopamine is assumed to reflect incomplete learning (and to inform the brain that more should be learned) rather than to control a motivational process. If dopamine is a teaching/learning signal, animals should perform less under reward uncertainty, as predicted by the Rescorla-Wagner model of learning (Rescorla & Wagner Reference Rescorla, Wagner, Black and Prokasy1972). Finally, incentive hope is likely to recruit brain structures not directly related to incentive salience, such as the dorsomedial striatum (Torres et al. Reference Torres, Glueck, Conrad, Moron and Papini2016). Although more research is required, incentive hope might be closely related to goal-directed behavior – for the processing that requires the dorsomedial striatum (e.g., Everitt & Robbins Reference Everitt and Robbins2005; Yin & Knowlton Reference Yin and Knowlton2006). Such a difference between incentive salience and incentive hope, and their respective brain structures, appears logical: The former process controls simple approach behavior for a goal reward, but the latter process might be involved in the search of a goal reward that is not there. Hope is unnecessary to approach a stimulus, but it may be necessary to seek it (Fig. 2).
Fig. 2. Interaction between incentive salience (“wanting”) and incentive hope. The left side depicts a situation where no uncertainty is present (the food items are predictable or accessible). Approach behavior results from incentive salience processes (involving the release of dopamine in the nucleus accumbens), while seeking behavior remains subactivated and, hence, prevented. The right side represents what happens when the animal is subject to food uncertainty (unpredictable access). The animal comes to produce not only incentive salience, but also incentive hope. Although approach could potentially be produced, its expression is canceled to the detriment of seeking, because the former behavior receives less activation than the latter. Of course, if uncertainty is temporarily abolished, then seeking is prevented and the stimulus is approached. This simple schema suggests that approach and seeking are differently processed while depending on the same motivational basis. It shows how a change in reward uncertainty can mechanically convert approach into seeking, and vice versa. DA = dopamine, GC = glucocorticoids, NAc = nucleus accumbens, DMS = dorsomedial striatum.
4.2.2. Beyond autoshaping: Extending incentive hope to natural context?
Incentive hope is not only about CSs, but also about rewards – just as incentive salience is. So, this process must be sensitive to reward density, a factor possibly more important than CS-related probabilities for animals living in the wild. Here, we are not interested in food density in itself, but rather in one of its major effects: A low density of food makes the variability in delays crucial for survival. When food density is high, the average delay to obtain a reward is short (Fig. 3A). Thus, the risk of starvation is low, and animals are not expected to produce incentive hope for quicker food. In contrast, when food density is low, the average delay to get rewarded is longer, causing a higher risk of starvation (Fig. 3B). Here, any delay shorter than the mean value offers more insurance for survival, so that a foraging individual is expected to develop incentive hope for quicker food. In a sense, a low density of food looks like Pavlovian autoshaping under partial reinforcement, because there is a possible absence of reward (many trials in a foraging bout may be unsuccessful) and uncertainty is unavoidable (no other environmental option exists). An example of food scarcity is the presence of grazing lawns in Africa. Grazing lawns are traditionally believed to be locally abundant and relatively permanent. But they are not permanent. Grazing lawns are predictable in space, but not in time, because the weekly pattern of rainfall is a stochastic process (Bonnet et al. Reference Bonnet, Fritz, Ginoux and Meuret2010). For this reason, herbivores cannot develop a perfect knowledge of the available resources, delaying food consumption in an unpredictable way. In this context, animals are likely to develop a high motivation for food obtained following short delays.
Fig. 3. (A) When food is abundant, there is no risk of starvation because the mean delay (the straight lines) to find edible items is short. (B) When food is scarce, there is a risk of starvation because the mean delay (the straight lines) to find edible items is longer. In such an environment, animals should hope for delays shorter than the mean and act accordingly.
It is assumed that the unpredictability of short delays when food density is low stimulates food seeking similarly to the probabilistic uncertainty of reward delivery in autoshaping; developing incentive hope for quicker food and CS reliability under a low density of food should therefore lead to an intensification of seeking behavior (leading to a reduction in potential delays and to a track of CS reliability) and to a longer search (increasing the chance that the effort deployed is profitable). Thus, the survival advantages of incentive hope are more visible in natural settings than in autoshaping, where the effort deployed has no functional consequences.
The idea that variability in delays in the wild may produce incentive hope does not contradict the suggestion that variable-delay schedules are preferred to constant-delay schedules because of quicker food delivery in the lab (see sect. 3.4). Incentive hope is not impossible with those schedules (animals may hope for quicker food), but it is less likely because the conditions of its occurrence are not fully met: There is no real uncertainty. The individual is rewarded on each trial, whether the constant or the variable option is chosen. This situation differs not only from natural context, but also from partial reinforcement in autoshaping, where a significant proportion of trials are not rewarded. In most choice schedules, the incentive hope hypothesis is superfluous because it makes the same prediction as the incentive salience hypothesis: Animals prefer variability only if it is associated with relevant properties – whether they hope or do not hope for those properties. A noticeable exception might be contrafreeloading, where animals under low food deprivation work to reduce environmental uncertainty – they seem to hope for exploitable information. But the predictions of the two hypotheses basically differ when variability is unavoidable, as in serial autoshaping and in the wild, because here the incentive salience hypothesis has to presuppose that behavioral invigoration or lengthening is due to the attractiveness of variability. But this contradicts the evidence that, in choice schedules, variability is not attractive in itself (e.g., McDevitt et al. Reference McDevitt, Dunn, Spetch and Ludvig2016). This means that incentive salience is not fully appropriate to explain behavior when uncertainty is unavoidable. Figure 4 summarizes the conditions maximizing the chance that incentive hope is produced.
Fig. 4. The incentive hope hypothesis. On top, the three conditions required for the development of incentive hope, which can be shown for different survival-related parameters (especially CS reliability, short delays, and additional information). Incentive hope is believed to increase food-seeking behavior and, when food is in sufficient amounts in the environment, food consumption as well. As a result, the animals seeking food items whose uncertainty is unavoidable have the opportunity to increase their fat reserves and/or to hoard more items. Autoshaping consists of a special case, in which incentive hope is only produced for CS reliability, and the experimental procedure does not allow the animal to increase reward rate. CS = conditioned stimulus.
As with probability, uncertainty in delays when food density is low might cause some stress in foraging animals, but it also enhances food seeking and food consumption compared with exposure of less motivated individuals to a safe environment. On the assumption that this view is correct, it may have strong functional implications for behavioral ecology. Our view suggests that the low motivation to forage when food is in safe density is an adaptation to remain fast and agile to escape from predatory attacks. By contrast, the higher motivation to forage when food is unpredictable is an adaptation to get the energy required to stay alive. This approach to fat regulation is in agreement with the functional interpretations proposed by behavioral ecologists, while shedding light on the causes that may underpin foraging.
5. Incentive hope can increase fat reserves: Computational evidence
To formalize our theoretical ideas, we developed the computer model of a small bird foraging on bugs in lawns or in clearings. This model had already been used to test the effects of several parameters (handling costs, rest periods, food quality, initial fat reserves, and predation risk), as well as the effects of incentive hope, on food consumption and fat accumulation in a safe and an unpredictable environment (Anselme et al. Reference Anselme, Otto and Güntürkün2017). This set of simulations showed that a higher motivation to forage can lead foragers to survive longer, increasing food consumption and fat accumulation only within certain limits when the environment is unpredictable. In this article, the model is used to illustrate a point discussed earlier: The fat reserves of highly motivated foragers exposed to an unpredictable access to food will increase only if food is available in sufficient amounts.
In the model, a single forager followed a pseudorandom trajectory at a constant speed in a two-dimensional environment that contained CSs associated with food items UCSs (CSs+) and might also contain CSs alone (CSs–). The CSs+ and the CSs– had pseudorandom distributions in the environment, which offered 0.25 million possible locations (500 × 500) –the large majority of which were empty (without CS+ or CS–). The forager was able to detect CSs from a distance shorter than or equal to a detection radius (whose maximal value was fixed in advance) and to approach them with a probability higher than for any other direction once detected (whose maximal value was also fixed in advance). This meant that the forager could locally modify its direction to reach the detected CS. Because of space limitations, we present a nontechnical description here. A full description of the forager's properties (as well as the code used for its implementation) is available as an online supplement.
Briefly, in a safe environment, the forager could encounter some CSs+ while traveling. Environmental safety meant that all of the CSs were fully predictive of food and that there was no risk of starvation – the amount of food UCSs was equivalent to a predefined safety threshold (for details, see the online supplement). A food UCS was consumed when the forager came to occupy the same x, y coordinates as its CS. A consumed UCS (and its CS) disappeared and a new CS+ reappeared somewhere else in the environment, to maintain food uncertainty and density constant. The energy value of the item was temporarily stored in a short-term storage system (“stomach and gut”) and then transferred at a constant rate to a longer-term energy storage system (“fat reserves”). Fat reserves decreased constantly and gradually over time, because of the energy costs related to traveling, but also to prey handling, CS inspection, and rest periods. The energy (fat) level resulting from this trade-off between consumption and energy expenditure controlled the forager's hunger-induced motivation (or “wanting”): High fat reserves caused a low “wanting” value, and lower fat reserves caused a higher “wanting” value. These motivational fluctuations had a direct impact on food seeking through an alteration of CS detectability and approach behavior. “Wanting” had the effect of increasing the forager's detection radius and of increasing the probability of approaching a detected CS; “wanting” increased CS attraction.
In an unpredictable (or unsafe) environment, the CSs+ were pseudorandomly mixed with CSs– and there was a risk of starvation – the amount of food UCSs was lower than the predefined safety threshold (for details, see the online supplement). Contrary to CSs+, which could disappear and reappear anywhere else once inspected and the associated food consumed, the inspected CSs– maintained their pre-inspection location throughout. Because the foragers had a greater risk of energy shortfall here, foraging motivation depended on fat-related “wanting” and also on incentive hope. In the model, incentive hope was a consequence of CS unreliability (a CS might or might not be associated with a food item), delay variability, and food density (see Eq. 4 in the online supplement). Incentive hope magnified the effects of fat-induced “wanting,” increasing the forager's detection radius and the probability of approaching a detected CS. Motivational strength had no effect on the forager's traveling speed.
Here, we compared eight foragers seeking food in three distinct environments. First, the safe environment contained 800 CSs+ and 0 CSs– (then referred to as S-800). Second, the moderately unpredictable environment contained 200 CSs+ and also 200 CSs– (U-200). Third, the highly unpredictable environment contained 60 CSs+ and 60 CSs– (U-60). The safety threshold value was 800 in each environment. Thus, relative to the safety threshold of 800 CSs+, the two unpredictable environments were suboptimal in the sense that they contained significantly fewer CSs+ (4 times fewer in U-200 and 13.3 times fewer in U-60) and also contained CSs– that could attract the foragers without providing them additional energy. Each forager traveled a distance of 3,000 steps (one step = distance from one location to the next) in an environment that was 500 steps long and 500 steps wide. Figure 5 represents the portion of each environment traveled by the foragers. As predicted, they explored smaller portions of a safe environment (19%) than of an unpredictable environment (68% in U-200 and 56% in U-60). In doing this, the foragers increased their chance of finding food items, just as real birds travel longer distances to find food in the harsh winter (e.g., Daunt et al. Reference Daunt, Afanasyev, Silk and Wanless2006; Hiraldo & Donázar Reference Hiraldo and Donázar1990; Lovette & Holmes Reference Lovette and Holmes1995). All of the foragers were exposed to prey-handling costs and to mass-dependent predation risk (see the online supplement), which could force them to rest at certain times (for additional simulations, see Anselme et al. Reference Anselme, Otto and Güntürkün2017).
Fig. 5. Portion of an environment traveled depending on its density of food. (A) Safe environment. (B) Moderately unpredictable environment. (C) Highly unpredictable environment. The safe environment was less explored than the two unpredictable environments. In each environment, the colored squares represent the total number of squares crossed by the eight foragers (a square was colored when at least one of the eight foragers entered, and consisted of a surface of 50 steps × 50 steps).
All of the foragers started in the middle of the environment with the same level of fat reserves. Fat reserves remained relatively stable over the 3,000 steps in the S-800 environment, they gradually increased in the U-200 environment, and they gradually decreased in the U-60 environment (Fig. 6A). This result indicates that there was a limit from which food unpredictability could not be adaptively countered, even if moderate decreases in the safe density of food had positive effects on the ability to store fat. In winter, fatter foragers have a higher survival rate than leaner foragers (Gosler Reference Gosler1996), and our simulation was in accord with this fact; in particular, fatter foragers were able to travel a longer distance than leaner foragers in case of a prolonged period of famine (Fig. 6B). Our model revealed that, compared with those in the S-800 environment, the foragers in U-200 stored more fat (F(1,21) = 23.554, p = 0.000) and consumed more food items (F(1,21) = 25.645, p = 0.000), while the foragers in U-60 stored less fat (F(1,21) = 116.972, p = 0.000) and consumed fewer food items (F(1,21) = 138.093, p = 0.000; Figs 6C and 6D). The effect sizes were very large (fat reserves: ηp2 = 0.85; food consumption: ηp2 = 0.87). Figure 6E illustrates that a higher motivation to seek food was effective only when the reduced density of food remained within an acceptable range; extreme (probably unrealistic) motivational values could not compensate for a too low density of food. Finally, compared to the foragers in S-800, the foragers in U-200 were more exposed to mass-dependent predation risk and the foragers in U-60 were less exposed (Fig. 6F; U-200: F(1,21) = 27.378, p = 0.000; U-60: F(1,21) = 36.786, p = 0.000; ηp2 = 0.86). But the energy lost because of the frequent rest periods induced by a higher predation risk did not prevent the foragers in U-200 from consuming more food and accumulating more fat reserves.
Fig. 6. The beneficial effects of increased food seeking imply that food amounts remain within a biologically acceptable range. (A) Compared with foragers exposed to a safe environment (group S-800), more fat is stored over time in group U-200, but there is a gradual loss of initial fat reserves in group U-60. (B) When food is not available, the ability to survive for longer periods = is proportional to the amount of fat stored. (C) Overall group comparisons indicate that fat reserves were higher in U-200 and lower in U-60 compared with S-800 foragers. (D) The number of food items consumed was higher in U-200 and lower in U-60 compared with S-800 foragers. (E) Motivational strength in seeking food was higher in U-200 than in S-800 foragers, but it even reached a greater intensity in U-60 foragers. (F) Mass-dependent predation risk was higher in U-200 foragers (because they gained weight) and lower in U-60 (because they lost weight) compared with S-800 foragers. Each data point on the abscissa must be multiplied by 50 to obtain the number of steps actually traveled (total: 60 × 50 = 3,000 steps).
By design, motivational strength was systematically higher in the unpredictable environments. What we have tried to demonstrate here is that a higher motivation in an unpredictable environment is theoretically sufficient to allow animals to consume more food and to store more fat reserves than when the environment is safe. This is not a trivial claim because it is not guaranteed in advance that motivation can compensate for a significant reduction in food probability and density. In other words, our simple simulation suggests that interpreting fat regulation in motivational terms is a plausible scenario.
6. Major predictions of the incentive hope hypothesis
Until now, the incentive hope hypothesis has been discussed on the basis of the existing data that may support it. However, the hypothesis can be satisfactory only if it is empirically testable by means of new predictions. In this section, we present a few predictions related to autoshaping or based on other methods of investigation, whether they refer to the lab or to the field. The new predictions are contrasted with those from other theories when possible.
1. The injection of a dopamine antagonist into the dorsomedial striatum should abolish the higher asymptotic response rate under partial reinforcement, but have no effect on (or amplify) the expression of negative contrast. However, frustration theory predicts that both types of responses (asymptotic rate and negative contrast) are related to frustration, so that a dopamine antagonist should affect them in a similar fashion.
2. Animals with a higher dopamine release should show higher asymptotic response rates under partial reinforcement and no negative contrast, whereas those whose brains release less dopamine should not evince higher asymptotic response rates under partial reinforcement but should express negative contrast. Strains of rats that differ motivationally or emotionally could be used here (e.g., Flagel et al. Reference Flagel, Robinson, Clark, Clinton, Watson, Seeman, Phillips and Akil2010; Sanna et al. Reference Sanna, Bratzu, Piludu, Corda, Melis, Giogi and Argiolas2017). Frustration theory predicts that the two types of responses will be obtained in one strain, but not in the other.
3. Individuals maintained under unpredictable deprivation periods in their home cage should show a higher break point in a progressive-ratio schedule than individuals that receive constant amounts of the same food every day in their home cage.
4. Late in training under partial reinforcement, the increase in responding should be observed whether the previous trial was rewarded or not (because the hope for reward on the next trial is independent of what was received just before). In contrast, frustration theory predicts that the increase in responding should only occur after a nonrewarded trial (because of frustration drive).
5. In small passerines, corticosterone-implanted birds should have higher dopamine levels in the nucleus accumbens and increase the intensity and/or the duration of their foraging bouts, compared with saline-implanted birds.
6. In small passerines, higher dopamine levels in the nucleus accumbens should boost food consumption and/or food-caching behaviors when the available amounts of food are sufficient.
These predictions are in keeping with the next priorities in the study of food seeking in animals. There is a vast literature on autoshaping, and a growing number of studies examine the neuronal basis of this process (e.g., Flagel et al. Reference Flagel, Cameron, Pickup, Watson, Akil and Robinson2011a; Reference Flagel, Clark, Robinson, Mayo, Czuj, Willuhn, Akers, Clinton, Phillips and Akil2011b; Hart et al. Reference Hart, Clark and Phillips2015; Saunders & Robinson Reference Saunders and Robinson2012; Sunsay & Rebec Reference Sunsay and Rebec2014; Torres et al. Reference Torres, Glueck, Conrad, Moron and Papini2016). In this context, it is important to test theoretical views that could help understand animal foraging in the wild, especially through an investigation of the neuronal correlates of response invigoration relative to ambiguous CSs. It could also be valuable to determine the behavioral effects that pretraining under uncertainty autoshaping may have on foraging behavior in more realistic ecological conditions. Conversely, understanding the motivational, emotional, and cognitive aspects of animals foraging is paramount (e.g., Bateson & Kacelnik Reference Bateson and Kacelnik1995; Cabanac Reference Cabanac1992; Dukas & Kamil Reference Dukas and Kamil2000; McNamara & Houston Reference McNamara and Houston1985; Pravosudov & Smulders Reference Pravosudov and Smulders2010). These psychological mechanisms must complement, not replace, the functional views suggested by behavioral ecologists and be compatible with the biological findings (e.g., about hormones and fat deposits) revealed by physiologists.
7. Implications of the incentive hope hypothesis
Modern Western societies are very demanding, in terms of both professional successes and social status. The demands are difficult to achieve because many people have to compete for short-term contracts of employment and have salaries that do not always reflect how hard the work is. As a result, successes in life depend partly on chance. This situation strongly contributes to the reasons so many people suffer from stress problems. As shown, stress favors the release of mesolimbic dopamine through an elevation of glucocorticoid levels in rodents (Barrot et al. Reference Barrot, Marinelli, Abrous, Rougé-Pont, Le Moal and Piazza2000; Piazza et al. Reference Piazza, RougePont, Deroche, Maccari, Simon and LeMoal1996; Rougé-Pont et al. Reference Rougé-Pont, Deroche, Le Moal and Piazza1998), a process also assumed to occur in birds, and incentive hope could result from utilization of the extra dopamine. We think that this process may help explain a number of pathologies such as drug addiction, problem gambling, and obesity. We show how the causes of uncertain-reward seeking (leading to adaptive responding in nature) may have maladaptive implications beyond the context in which natural selection operated initially, because of the disappearance of their functional relevance.
7.1. Drug addiction
Animal and human research indicates that social and environmental stresses make individuals more vulnerable to the addictive properties of drugs of abuse and also more prone to attribute motivational salience to CSs (Beckmann & Bardo Reference Beckmann and Bardo2012; Diaz et al. Reference Diaz, Siontas, Mendoza and Arvanitogiannis2013; Lomanowska et al. Reference Lomanowska, Lovic, Rankine, Mooney, Robinson and Kraemer2011; Nader et al. Reference Nader, Chauvet, Rawas, Favot, Jaber, Thiriet and Solinas2012; Pattison et al. Reference Pattison, Laude and Zentall2013). Also, developmental stress boosts locomotor activity in starlings (O'Hagan et al. Reference O'Hagan, Andrews, Bedford, Bateson and Nettle2015), though not their speed to respond to ambiguous stimuli (Bateson et al. Reference Bateson, Emmerson, Ergün, Monaghan and Nettle2015). Our view suggests that incentive hope is an adaptation shaped by natural selection to reduce the risk of starvation when food is in short supply. Incentive hope can be a product of evolution because it is effective in the wild, allowing small birds to cache more food and/or to store more fat. The human environment meets the conditions for the recruitment of incentive hope (reward “wanting,” occasional failures, and unavoidable uncertainty). But this motivational process may fail to do its job here, because the ability to escape from the stressful contexts is often independent of the individual's willingness to change the situation (e.g., poverty, repeated bad luck, constraining work conditions). A higher sensitivity to drugs of abuse under chronic stress might be a response to the apparent need for more dopamine. Dopamine's stimulating effects may indeed lead to actions and biased perceptions that give the impression of greater opportunities to control the events of one's life. In traditional societies, when shamans fall into a trance after taking some drugs, they call on supernatural powers to solve stressful situations such as curing disease and defeating the enemy. Local people are convinced that this strategy will help them deal with adversity. However, this response turns out to be maladaptive in modern Western society, because most of the problems encountered cannot be “solved” that way: Taking drugs repeatedly often has the effect of degrading (instead of improving) the individual's socio-professional life.
A more effective strategy consists of engaging in activities that reorient attention toward new, stress-free objectives. For example, drug-dependent rats reared in a socially enriched environment and/or an environment that contains opportunities to do various activities significantly reduce their drug consumption and even stop showing neural sensitization of their dopamine neurons (Bardo et al. Reference Bardo, Klebaur, Valone and Deaton2001; Cosgrove et al. Reference Cosgrove, Hunter and Caroll2002; Lespine & Tirelli Reference Lespine and Tirelli2015; Nader et al. Reference Nader, Chauvet, Rawas, Favot, Jaber, Thiriet and Solinas2012; Solinas et al. Reference Solinas, Chauvet, Thiriet, El Rawas and Jaber2008). The incentive hope hypothesis provides an original view of neural sensitization, which is viewed as an adaptive strategy of the brain to make dopamine neurons more responsive to situations in which stress persists a long time. In a natural context, this process increases the chance of managing those situations, while its “hijacking” to face modern Western-society problems is no longer effective and contributes to the development of addictive behaviors.
7.2. Pathological gambling
The link between human gambling behavior and dopamine release has been empirically established (Joutsa et al. Reference Joutsa, Johansson, Niemelä, Ollikainen, Hirvonen, Piepponen, Arponen, Alho, Voon, Rinne, Hietala and Kaasinen2012; Linnet et al. Reference Linnet, Mouridsen, Peterson, Møller, Doudet and Gjedde2012), but the causal and functional explanations of this process remain unknown. Like drug addiction, problem gambling may be the consequence of chronic stress in a demanding societal context. For example, electronic gambling machines, which are associated with potentially large payout after a short delay on each trial, are the favorite game of problem gamblers who try to escape stressful situations in their life (Nower & Blaszczynski Reference Nower and Blaszczynski2010; van Holst et al. Reference van Holst, van den Brink, Veltman and Goudriaan2010). Compared with non-gamblers, they are not interested in lotteries at all, which involve delaying gratification, and they are strongly motivated to earn additional incomes (Nower & Blaszczynski Reference Nower and Blaszczynski2010). They also differ from other categories of gamblers, such as horse racing and casino gamblers, who attempt to replace feelings of boredom with higher levels of arousal (van Holst et al. Reference van Holst, van den Brink, Veltman and Goudriaan2010). In electronic machine gamblers, the gradual sensitization of corticosterone-induced dopamine release over repeated exposure to gambling opportunities should have contributed to the development of incentive hope more than in non-gamblers and occasional gamblers. It is predicted that electronic machine gamblers will report higher hopes for money than non-gamblers and occasional gamblers before a trial or before placing a bet. It is also predicted that hopes will be higher than any negative effects such as frustration and stress. Why do people gamble, then, if, as suggested earlier, uncertainty is not sought for itself? The reason is that casinos act similarly to autoshaping boxes: they consist of a confined environment in which any outcome is uncertain. Such an environment favors incentive hope in vulnerable individuals and hence invigorates and lengthens the propensity to seek (monetary) rewards. The fact that a casino is an artificial environment (just as an autoshaping box is) does not abolish seeking behavior, which is genetically fixed in our mammalian brains – for the same reason, captive passerines may downregulate their fat reserves despite the absence of actual predators (Verdolin Reference Verdolin2006). We hypothesize that pathological gambling is the consequence of a behavior that was adaptive for our ancestors (increased reward seeking when the environment was unpredictable) but has ceased to be adaptive in our modern Western societies (Anselme & Robinson Reference Anselme and Robinson2013).
7.3. Obesity problems
Today, many people suffer from overweight. The genes that control fat storage were also present in our ancestors, but their expression came to be problematic only very recently. The reason is that fat-rich food is cheap and easy to access, allowing people to eat more than they really need (Bodor et al. Reference Bodor, Rice, Farley, Swalm and Rose2010; Hill & Peters Reference Hill and Peters1998). However, food consumption is also partly associated with food insecurity in women with a low-economic status living in rich countries (Nettle et al. Reference Nettle, Andrews and Bateson2017). This means that economic uncertainty (irregular, low incomes) in a demanding society may lead to behaviors that favor the accumulation of fat reserves in humans, as observed in small passerines and other species. Fat accumulation was certainly adaptive in ancestral human societies, in which rich food (meat, honey, etc.) was rare and required a lot of work to be obtained. But this situation leads to overweight in modern humans, for whom junk food can be found at every street corner. Sinha and Jastreboff (Reference Sinha and Jastreboff2013) pointed out the influence of glucocorticoids and corticotropin-releasing factor (CRF) on dopaminergic transmission, which may increase motivation for highly palatable food and consequently may promote changes in body fat mass (see also Corwin Reference Corwin2011). The incentive hope hypothesis is compatible with this view and provides a theoretical framework to explain the very nature of uncertainty-induced motivation. To reduce its maladaptive consequences, a first direction to follow should be to decrease the risk that people experience adversity in life, for example, through education, employment security, good salaries, and well-being at work. But more practically, having fixed feeding routines should increase the feeling of food security and contribute to maintain appropriate fat reserves (Nettle et al. Reference Nettle, Andrews and Bateson2017). A second direction to follow would be to reduce the temptations, which bias the perception of what people really need. For example, the development of local shops – as opposed to large shopping centers – points in this direction.
8. Conclusion
In this article, we argued that fat regulation essentially depends on how much animals are motivated to consume food. In addition, we showed that this mechanistic approach is computationally tenable. The concept of incentive hope may account for the evidence that birds and mammals respond more to reward-related cues in situations in which uncertainty cannot be avoided. Incentive hope is related to incentive motivation, but irreducible to the concept of incentive salience for at least two reasons. First, the incentive salience hypothesis does not capture the motivational effects of reward uncertainty and therefore cannot explain them. Second, if incentive hope was only some additional incentive salience under uncertainty, animals should prefer an uncertain or variable option over a certain or constant option in a free-choice task. But we showed that uncertainty is not attractive in itself. Incentive hope is also different from frustration and prediction error. This new concept may have profound implications for the understanding of animal (and perhaps human) behavior, in the sense that many aspects of reality are uncertain in essence. Nevertheless, more thorough investigations are necessary with respect to the psychology and neuroscience of animal foraging. Outstanding questions might be the following: Does the activity of dopamine neurons sustain the duration and the intensity of foraging? Which are the respective roles of the ventral and dorsomedial striatum in foraging? How do CSs impact foraging? What is the relative importance of CSs and food density in controlling foraging activity?
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0140525X18000948
Acknowledgments
This work was supported by the Deutsche Forschungsgemeinschaft through An1067/1-1 to Patrick Anselme and through Gu227/16-1 to Onur Güntürkün. We thank Tobias Otto for his great support with the foraging model.
Target article
How foraging works: Uncertainty magnifies food-seeking motivation
Related commentaries (22)
A neural basis for food foraging in obesity
Beyond uncertainty: A broader scope for “incentive hope” mechanisms and its implications
Beyond “incentive hope”: Information sampling and learning under reward uncertainty
Complex social ecology needs complex machineries of foraging
Considerations for the study of “incentive hope” and sign-tracking behaviors in humans
Does the “incentive hope” hypothesis explain food-wasting behavior among humans? Yes and no
Extending models of “How Foraging Works”: Uncertainty, controllability, and survivability
Food security and obesity: Can passerine foraging behavior inform explanations for human weight gain?
Food seeking and food sharing under uncertainty
Food-seeking behavior has complex evolutionary pressures in songbirds: Linking parental foraging to offspring sexual selection
Foraging extends beyond food: Hoarding of mental energy and information seeking in response to uncertainty
Hoarding all of the chips: Slot machine gambling and the foraging for coins
Hope, exploration, and equilibrated action schemes
How uncertainty begets hope: A model of adaptive and maladaptive seeking behavior
Mechanistic models must link the field and the lab
Overlapping neural systems underlying “incentive hope” and apprehension
Random isn't real: How the patchy distribution of ecological rewards may generate “incentive hope”
Simulating exploration versus exploitation in agent foraging under different environment uncertainties
The value of uncertainty: An active inference perspective
Unpredictable homeodynamic and ambient constraints on irrational decision making of aneural and neural foragers
“How Foraging Works”: Let's not forget the physiological mechanisms of energy balance
“Incentive hope” and the nature of impulsivity in low-socioeconomic-status individuals
Author response
Incentive hope: A default psychological response to multiple forms of uncertainty