A. Defining the problem: Markov decision processes

Before defining MDPs in abstract terms, it is helpful to orient ourselves around the concrete details of an idealized example. Imagine a rat, Lenny, in a maze with cheese and electric shocks (Figure. 1). Lenny must decide how to traverse the maze in a way that achieves his objectives. As described above, we assume that the subjective goodness or badness of the cheese and electric shocks can be put on a common numeric scale, which in the MDP framework is called “reward.” For example, Lenny’s rewards might be: CHEESE = +1, SHOCK = -1. Lenny, then, has a single objective: maximize the long-term accumulation of reward. Lenny could then compute the amount of long-term reward gain expected from turning “left” or “right” at each fork in the maze, and simply choose the option with the highest amount.

Figure 1. MDP representation of a simple decision problem. A rat must decide how to traverse a maze (represented as a set of 6 states, each with a set of potential actions) to maximize long-term accumulation of reward. Rs represent the intrinsic rewards associated with each terminal state, and Qs represent the average value (i.e., long-term expected future reward) that the rat would learn to associate with each prior action. For instance, turning right at State 1 eventually leads to the cheese in State 6, and the rat would therefore learn that Q(state 1, action R) = +1.

This situation — an agent has a set of well-defined choices with well-defined consequences, and her goal is to maximize a single numerical objective — is what MDPs formally describe. MDPs carve the world into a set of discrete states S, which can be either decision points or terminal nodes. The decision points each have a set of actions A _s. (At terminal nodes, there are no more actions to be taken [e.g., the end of the maze], so A _s = ø.) The decision points capture all the moments where the person could do meaningfully different things. For instance, Lenny could carve the maze up into the states in Figure 1, where each fork is a decision point with the actions “left” and “right.”

But note that this carving is subjective: S and A _s are not objective features of the external environment, but variables in the person’s head. Lenny could have incorporated the temperature of the room and made separate decision points for “initial fork in hot room” and “initial fork in cold room.” Or he could have codified each physical step as a decision point, with the actions “move left foot forward,” “move right foot forward,” “move left foot backward,” and so on. States and actions can also be specified at various levels of abstraction.^{Footnote 24} Lenny could conceptualize his actions as “turn left” or “turn right,” but he could also conceptualize them as “turn body 27 degrees counterclockwise,” “move left foot 1.36 centimeters forward,” and so forth. These variables are subjective and, therefore, could be influenced by norms.

MDPs also formalize the consequences of choice. There are two questions to answer after making a choice: What happened, and how good or bad was it? In MDPs, after selecting an action at a decision point, an agent transitions to a new state (either another decision point, or a terminal node). This process is captured by a noisy transition function T(s, a, s’), which gives the probability of transitioning to any new state s’ after choosing an action a at a decision point s. This is the answer to the question, “What happened?”

Then, the agent receives a reward. In principle, how much the agent likes or dislikes the consequences of a choice could be based on all three variables: where the agent was (s), what the agent chose (a), and where the agent ended up (s’). Think of a daredevil jumping a canyon on a motorcycle. Her reward from this experience will depend on where she was (at the edge of a canyon), what she did (ride a motorcycle forward), and where she happened to end up (on the other side, or at the bottom of the canyon). Thus, to relate the person’s objective to the consequences of choice, we introduce a reward function R(s, a, s’), which gives the amount of reward a person receives for taking action a at decision point s and ending up at state s’. Note that, just like the previous representations, the reward function is subjective: Cheese and shocks are only good and bad in someone’s head. Thus, the specification of subjective rewards could also be manipulated by norms.

So MDPs describe decision situations with four variables: the set of states S, the actions at each decision point A _s, the transition function T, and the reward function R. In plain English, the person needs to know what the states of the world are, what she can do at each state, what next state she will arrive at, and how good or bad the consequences are. All four of these variables could, in principle, be influenced by norms in a way that would promote compliance.^{Footnote 25}

B. Discovering solutions: Reinforcement learning

Reinforcement learning is a branch of computer science that seeks algorithms for learning and deciding within MDPs. At the most abstract level, the optimal approach is for an agent to compute the value of each action available at each state, which has a precise definition: the expected sum of future rewards conditioned on choosing that action at the current decision point. Then, the person can choose the action with the highest value. This is analogous to computing the expected utility of a gamble in economic decision theory.^{Footnote 26}

Although value can be defined simply, calculating it is complicated. In a sequential setting such as the MDP, the long-term rewards following from an agent’s present decision depend not only on that decision, but also on the subsequent decisions that the agent makes. In other words, the agent needs to know something about itself — for a chess algorithm to know which piece to move, it needs to consider what subsequent moves it is likely to implement as the game unfolds. This can be formalized via the concept of a “policy,” which is a complete probabilistic mapping from state to action; a description of the likelihood of the agent doing each possible thing in each possible set of circumstances. Thus, an agent’s assignment of value to an action must be conditioned not only on the agent’s present state but also on its own policy. From this vantage point, the agent’s overarching goal is to develop an optimal policy.

A consequence of this complexity is that rational decision-making is often intractable. Consequently, much work in reinforcement learning seeks to find computationally efficient mechanisms for approximating value representation. Broadly speaking, many algorithms can be positioned on a spectrum from those that are highly rational (achieving accuracy at the cost of computation) to those that are highly heuristic (sacrificing accuracy to lower computational demands). This distinction forms a major dividing line between rival theories of norm compliance, and so it worth our careful attention.

The standard economic approach defines the rational end of this spectrum: Use an explicit model of the situation to perform the exact calculations. For example, if Lenny knows that turning left twice leads him to cheese and turning right twice leads him to shock (that is, he has a representation of the transition function T), he can compute that the expected value of turning left at the initial fork is +1. In MDPs, this is known as goal-directed planning, and captures the kind of forward thinking often associated with human rationality.^{Footnote 27} In the RL literature these methods are typically described as “model-based,” because they require an agent to learn and explicitly represent a model of the transition structure of the environment, which forms the cognitive basis for planning. This approach can be very precise — even perfectly so — but it can also entail major computational costs.

Alternatively, Lenny could approximate the value of turning left at the fork by keeping a running average of the reward he’s received after taking that action in the past. In this case, he wouldn’t turn left because he believed it would lead to cheese; he would turn left simply because it had been a good choice in the past. In other words, Lenny would be acting habitually. This backward-looking method is often associated with irrationality^{Footnote 28} — or, more charitably, with heuristic cognition^{Footnote 29} — and there is extensive evidence that people maintain and rely on these “cached” averages of past reward.^{Footnote 30} In the RL literature, these methods are typically described as “model-free,” because they do not require learning, representing, or planning over a model of the transition structure of the environment; they just rely on averages of past reward. This approach often sacrifices precision but achieves major computational savings. We denote Q(s,a) as the average value an agent has obtained from choosing action a in decision point s in the past. Q’s are the fifth and final decision variable in the RL framework that could be influenced by norms.^{Footnote 31}

The computational distinction between “model-based” and “model-free” approaches — that is, between habit and planning — helped to crystalize a more fluid set of concepts that have long been fundamental to behavioral research. Psychologists since Thorndike^{Footnote 32} have debated the extent to which complex behavior is the result of planning over an internal representation of the causal structure of the world, or instead the accumulation of adaptive heuristics shaped under the prior history of reward and punishment. RL offers one means of describing the competition between rival cognitive “systems” that are characterized by different tradeoffs of accuracy against computational demand.^{Footnote 33}

The conflict between habit and planning is spelled out in a simple behavioral paradigm that mirrors our example of Lenny. A rat is trained to press a lever in order to obtain a food reward. Then, it is removed from the box and given free access to as much of its favorite foods as it wants, until it is completely sated and shows no further interest in the food. Finally, the sated rat is returned to the box with the lever. If the rat had received small amounts of training, it will not push the lever because it knows it doesn’t want the food (that is, it engages in planning). However, if it was overtrained and formed a habit of pressing the lever, it will move to the lever and continually press it, ignoring the food that accumulates on the floor.^{Footnote 34} The irrational persistence in an action that was previously reinforced is the hallmark of habitual control of behavior. Reinforcement learning offers a means of making this distinction between habit and planning computationally precise, and assimilates a range of previously disparate decision-making models into a common framework.^{Footnote 35}

D. Summary

Together, MDPs and RL offer a powerful model of decision situations, with five variables (S, A _s, T, R, and Q) that could be influenced by norms (Figure 2). This framework will form the backbone of our discussion. But it is important to emphasize again that this is not the only viable model of decisions. There are, for example, many models that forgo the notions of reward and value, and don’t assume all objectives can be mapped onto a common currency.^{Footnote 36} We adopt a reward-based framework because it is widespread in both cognitive decision theory and theories of norms, but future work might pursue these alternatives. There are also models of norm compliance that forgo any notion of mental representations at all.^{Footnote 37} While nonrepresentational models have much to contribute, our goal is to see how much traction cognitive models can gain.

Figure 2. Five cognitive decision variables that norms could influence, and the theory of norm compliance that corresponds to each variable. On the folk model, norms change the decision maker’s internal causal model of the world (“if I cheat on my taxes, I’ll go to jail”). On the habit model, norms change the stored values of options via day-to-day reward and punishment of obedience or disobedience (“every time I cheated on things like taxes in the past, bad things happened”). On the internalization model, norms affect the intrinsic reward that people assign to different outcomes (“it’s wrong to not pay your fair share of taxes”). And on the unthinkable action model, norms change which actions are even considered (“I didn’t even think to cheat on my taxes”).

A. Compliance by planning: A folk theory of norm influence

The simplest theory of norm influence was stated by Glaucon in Plato’s Republic: People “are only diverted into the path of justice by . . . force.”^{Footnote 38} Paraphrasing: People follow norms because they fear reprisal, or a loss of reputation. Although Glaucon’s specific theory focused on punishment, we are concerned not with the content of people’s expectations (“I’ll go to jail”), but rather the form of the influence upon decision-making. The claim is that people obey norms because they explicitly represent the likely outcomes of compliance versus noncompliance, and conclude that the likely outcomes of compliance hold a greater prospect of maximizing long-term reward. Thus, this theory encompasses a wide range of specific motivations: Avoiding jail, earning respect, winning friends, maximizing profit, and so on. This view is common enough to be called the folk theory of norm influence, and is the launching point for our discussion.

Construed within the RL framework, the folk theory is comprised of two key claims:

1. 1. The existence of the norm changes the agent’s expectation that future reward will be maximized via norm compliance, by altering the agent’s internal representation of the transition function T.

2. 2. The existence of the norm does not change the agent’s assignment of reward (that is, the reward function R). In other words, norm compliance has value to the agent exclusively because it is linked to other rewards that the agent already holds (such as food, pain, and so forth).

In the sections that follow, we will review alternative models of norm compliance that depart from the folk theory in two different ways. One claims that norms influence behavior not by planning over an explicit model of future reward, but rather by heuristics and habits established by the prior history of reward (that is, norms affect Q). The other claims that norms change not just our expectation of reward but rather the very set of events that we find rewarding at all (that is, norms affect R, for example, by “internalization” of the norm as an intrinsic good). Thus, the folk theory stands apart from these alternatives principally in its claims of planning over an unmodified specification of reward.

It is uncontroversial that the folk theory captures at least part of the manner in which norms influence human behavior. At least some of the time, people pay taxes because they fear an audit; vote because they want to influence the outcome; tip because they want good service; recycle because the want to burnish their social image; and so on. But it is equally uncontroversial that the folk theory is incomplete. The problem is that people routinely obey norms even when they know there is little chance of reprisal. They report their taxes honestly when the risk of an audit is incredibly low; vote in elections that they have no chance of influencing; tip in restaurants they will never revisit; recycle, even when living alone; and so on.^{Footnote 39}

The persistence of norm compliance in the absence of explicit expectations of enforcement has been most carefully demonstrated with laboratory economic games.^{Footnote 40} In these games, people are given an endowment of bonus money and asked to make decisions that will affect both their own payment and the payment of other subjects. For example, people are sometimes asked whether they want to contribute to a public pool that will benefit everyone else but cost them personally. Other times they are asked to split the endowment with another subject. Time after time, most participants at least sometimes obey norms prescribing fairness, cooperation, reciprocity, or honesty. Importantly, they follow the norms even when the game is one-shot and anonymous — in other words, when noncompliance is guaranteed to go unpunished.^{Footnote 41}

We call this phenomenon — people’s dogged persistence in obeying norms — the persistence problem. People don’t always persist in norm compliance, but they do it often enough to make us question whether explicit expectations of reward are truly the driving force. The folk theory can be thought of as a null hypothesis, and the persistence problem the data that force us to consider the more complex alternatives that follow.

There are a few concerns with this analysis that will be useful to address before moving to more complex theories. First, is the persistence problem really a problem, or just an ordinary mistake? People could simply be failing to notice that they are in a situation where punishment doesn’t apply, or fail to incorporate this knowledge into their representation of the transition function.^{Footnote 42} Though this objection is difficult to rule out in real-world examples, the simplicity and transparency of countless laboratory experiments makes it unlikely that subjects are not representing the contextual one-shot anonymity.^{Footnote 43} Moreover, people do often incorporate this information into their planning, increasing compliance when people are observing them or can reciprocate.^{Footnote 44} The interesting fact is that their levels of norm compliance are far from zero even in the one-shot anonymous case.

Second, we have emphasized examples like tax reporting, recycling, and one-shot economic games to draw inferences about norm compliance. But how do we know that these are actually examples of norm compliance? People might have performed these behaviors in the absence of exposure to any norm (for example, because of evolved instinct). Again, while this is difficult to rule out in the real world, numerous experiments show that norms drive people’s behavior in anonymous situations. People exhibit enormous cross-cultural variation in one-shot economic games, and this variation is partially explained by differences in cultural norms.^{Footnote 45} And manipulating norms changes behavior. People walking alone rarely litter after seeing anti-littering signs, but litter quite often after seeing piles of trash on the ground.^{Footnote 46} People reuse hotel towels more often, and give away more money in economic games, when told it is the norm.^{Footnote 47} All this suggests that people are influenced by norms, even when they won’t be sanctioned for disobedience — and thus the folk theory of norm compliance is incomplete.

To motivate our exploration of the theory space, we will repeatedly draw on these kinds of examples as genuine instances of norm-following. The key assumption in each instance is that the aggregate behavior — a tendency to offer fair splits of a monetary endowment, contribute to a public good, return an investment, and so forth — is in part driven by prior exposure to norms. Thus, from patterns in the behavior, we can infer features of how norms influence us. In many cases, there is experimental (and anecdotal) evidence to support this assumption. But in other cases, it has not been explicitly tested.^{Footnote 48} We will highlight claims that demand more rigorous experimental support from future work.

Finally, it is important to note that our review of literature will be dramatically incomplete. We focus on studies that we believe offer the most precision and insight into the cognitive underpinnings of compliance, and, in doing so, pass over much important work in sociology and other fields. We hope to offer a cognitive perspective that naturally complements these alternative approaches.

B. Compliance by habit: Heuristic approaches to value approximation

The folk theory assumes that people are representing their environment’s transition function (“if I turn right at the fork, it will lead to cheese”), and selecting actions via forward planning (“I want cheese, therefore I should turn right”). They know the consequences of their choices, and act on that knowledge. This assumption is what causes trouble for the folk theory: People seem to follow norms even when they know the typical motivating consequences are suspended (for instance, in one-shot anonymous settings).

There are two ways out of this bind. Perhaps the consequences that motivate norm compliance are not sanctions or reputation, but an intrinsic “social preference,” such as promoting fairness or being a good person. This solution to the persistence problem depends on altering the reward function R, and it is pursued in the next section.

First, though, there is a simpler alternative. As described above, people don’t always plan ahead; they often act out of habit. More formally, instead of representing the transition function T (that is, the explicit consequences of actions), they store the average value Q of past actions, and simply choose actions with higher Qs — actions that accumulated higher reward in the past. A habitual Lenny turns left in the maze, not because he knows it will lead to cheese, but because he remembers that it led to an average goodness of +1 in the past. If you told him that the cheese had moved, it wouldn’t affect his immediate decision-making, because he’s not even considering cheese when making his decision.

Similarly, a habitual human might propose a fair monetary split in an economic game, not because she knows it will avoid sanctions and improve her reputation, but because she merely remembers that being fair led to an “average goodness” of +1 in the past. If you told her that the game was one-shot/anonymous and she couldn’t be sanctioned, it wouldn’t affect her immediate decision-making, because she’s not even considering sanctions when making her decision.

We call this the habit theory of norm compliance. In everyday life, most situations are not one-shot or anonymous, and complying with norms is typically beneficial (to avoid sanctions, accrue reputation, and so on). This experience produces high-cached Q values for norm-compliant actions. Then, when people are in one-shot anonymous settings where the typical benefits are removed, they choose based on those Q values, which are not immediately affected by the knowledge of anonymity. Thus, people persist in norm compliance.^{Footnote 49}

One point in favor of the habit theory is that people appear habitually averse to causing direct personal harm, even in contexts where it maximizes welfare.^{Footnote 50} In contemporary societies, there are strong norms against causing typical forms of harm: pushing, hitting, shooting, stabbing, and so on. The habit theory predicts that people will therefore have developed a habitized aversion — that is, a low Q value — to performing those behaviors. In support of this, people are unwilling to perform these actions (for example, pushing someone in front of a runaway trolley), even in the presence of a causal model linking such an action to positive consequences (for example, saving five other people^{Footnote 51}). Moreover, people exhibit physiological signs of aversion to pretend actions that resemble acts of physical harm, such as pulling the trigger of a fake replica handgun while pointing it at a person’s head.^{Footnote 52} Like the rat who keeps pushing its food lever even when full, people seem to be relying on a cached value that they have assigned to certain norm-violating behaviors — even when the typical consequences of those behaviors are suspended.^{Footnote 53}

The habit theory of norm compliance is also supported by a suite of experimental results motivated by the fact that acting on habits requires less computation time/resources than planning. (Planning requires the agent to integrate over a potentially complex transition function, while acting on habits just requires a comparison among pre-computed Q values.) Thus, according to the habit theory, people with less time or mental resources should be more compliant.

Indeed, when people in economic games are put under time pressure or cognitive load, they become more cooperative,^{Footnote 54} fair,^{Footnote 55} reciprocal,^{Footnote 56} and—sometimes—more generous.^{Footnote 57} This finding resonates with real-world reports of extreme prosociality: People who risk their lives to help others almost always describe their decision process as quick and intuitive (like acting on a habit), not slow and reflective (like planning).^{Footnote 58}

Moreover, the effect of time pressure/cognitive load in economic games is dependent on prior experience: It is strongest for people who report positive cooperative experiences in their daily life,^{Footnote 59} and disappears with extensive experience in one-shot anonymous economic games.^{Footnote 60} Both of these findings are explained by the habit theory. Only people who were rewarded for complying with norms in the past should form a habit of it. And, while Q values are not immediately affected by contextual anonymity, they will be in the long run: repeated experience in a context where compliance is not beneficial will drive down the Q values until compliance is no longer a habit. Imagine we move Lenny’s cheese. If Lenny keeps habitually turning left in the maze, eventually the average reward for turning left will plummet, and Lenny will stop turning left. Similarly, if people keep playing one-shot anonymous economic games, their average stored reward for complying with norms will drop, they will stop complying out of habit, and thus the effect of time pressure will disappear. These experimental results therefore support the habit theory of norm compliance.^{Footnote 61}

In sum, habit theory likely explains some part of norm compliance. But it is also incomplete. The problem is that, even after people have enough experience with one-shot anonymous settings to lose their habit for compliance, they continue to comply. In the real world, experienced travelers still tip in foreign cities; experienced solo-hikers still avoid littering when alone in public parks; hotel guests still reuse towels on the tenth day of their stay. In all these cases, if people were acting out of a habit developed in non-anonymous settings, experience with the anonymous setting should eliminate compliance — but it doesn’t.

This phenomenon is most precisely demonstrated with laboratory games. As described above, people who participate in many economic games seem to lose their habit for compliance; they stop showing an effect of time pressure on compliance choices. Yet they continue to contribute to public goods, propose fair monetary splits, return investments, and so on, at almost the same rate^{Footnote 62} — it just no longer matters whether they have a short or a long amount of time to do so. The persistence of norm-compliant behavior even in the face of massive experience with one-shot, anonymous contexts is difficult to explain under the habit theory.

In sum, the habit theory appears only to explain “shallow” persistence: people comply with norms in the first few anonymous situations they encounter. But it does not naturally account for “deep” persistence: people continue to comply with norms, even after they have enough experience with anonymous situations to kick the habit. Norms seem to do more than just instill habits.

C. Compliance by internalization: Norms as sources of intrinsic reward

A popular idea, hinted at before, is that norm compliance may sometimes be valued intrinsically, rather than as an instrumental means of avoiding punishment or attaining reward. Through socialization, people internalize the norm and come to care about the things that norms prescribe. People are fair, not because they know unfairness would be punished, but because they care about producing fair outcomes.^{Footnote 63} They contribute to public goods, not because their contributions are being externally tracked, but because they believe it’s the right thing to do.^{Footnote 64} If this were true, then of course people would continue to comply with norms in anonymous settings — anonymity doesn’t remove the consequences they care about.

This concept can be made precise within the reinforcement-learning framework by positing that exposure to norms changes people’s reward functions, R.^{Footnote 65} Recall that the function of reinforcement learning is to estimate the instrumental value of an action based on the long-term expectation of intrinsic reward in subsequent states. Whereas the folk theory and habit theory posit that norms affect behavior by altering the way we perceive the instrumental value of compliance, the internalization theory posits that norms affect behavior by altering the very set of states or actions we find intrinsically rewarding.

This theory naturally accounts for deep persistence without requiring us to abandon the possibility of rational planning.^{Footnote 66} In the language of MDPs, the dictates of the norm (“split goods fairly”) have just been internalized as another part of people’s reward functions (for example, +1 for proposing a fair split in an economic game), which can be flexibly and rationally traded off against other parts of the reward function during planning. Internalization theory also explains the role of guilt (and other negative emotions) in preventing norm violations: guilt is part of the internalized negative reward for violating norms.^{Footnote 67}

Although the internalization theory can be stated in terms of MDPs, it is important to note that it falls outside the scope of the RL algorithms typically pursued in machine learning contexts. Ordinarily, a reinforcement-learning agent operates with an unchangeable reward function defined by the programmer. It should be obvious why: If the programmer specifies that winning chess is the reward for an agent, it would not want that agent to later redefine rewards around, say, making pretty patterns with the pawns. Yet this, more or less, is what internalization theory posits: that humans initially designed to maximize one thing (for instance, maximum food for themselves) might later autonomously decide to try to maximize something else (for instance, equal food for all). An interesting area of contemporary research seeks to understand why natural selection might favor such an architecture, in which reward itself is subject to evaluation and change.^{Footnote 68}

Although the internalization theory does some violence to traditional assumptions of reinforcement learning, its commitments can also be made much more precise within the RL framework. For instance, there is disagreement in the literature over what, precisely, gets internalized during norm learning. Some theories posit that humans learn to intrinsically value certain states of affairs, such as an equitable division of resources among people.^{Footnote 69} Other theories posit that humans learn to intrinsically value certain actions, such as the act of generating an equitable division.^{Footnote 70} These have been regarded as competitors, yet often the precise commitments of various theories are not transparent.

A theory of norm internalization couched in the language of MDPs is naturally committed to precise answers to all of these questions, simply by specifying a reward function. The reward function R takes as input the current state s, the chosen action a, and the subsequent state s’. If norm internalization made the outcome of compliance rewarding (us receiving an equal amount of money), then R would be positive for an s’ with equal splits, for any a.^{Footnote 71} If it made the act rewarding, then R would be positive for an a of proposing an equal split, for any s’.^{Footnote 72} If the reward was for specific norms, then R would be defined over a set of concrete actions a; if the reward was for following norms in the abstract, then R would be defined over an abstract action space of “follow the norm” or “don’t follow the norm.”^{Footnote 73} If context mattered, then R would be positive for only certain states s. MDPs can precisely model all of these situations, showcasing the breadth of ideas that fall under the umbrella of the reward theory of compliance.

D. A challenge to value-guided theories

Each of the theories that we have considered so far shares a common assumption: norms intervene on decision-making by altering the value of actions under consideration, as they compete against alternatives in a common currency. This would appear to be a basic commitment of any theory of norm compliance grounded in value-guided decision-making.

An important challenge to this perspective is that people sometimes disregard, or are highly insensitive to, the costs and benefits of complying with or violating these norms. For example, many people say there is no amount of benefit that would justify selling organs, auctioning off brides, or mandating abortions to limit population growth.^{Footnote 74} If a hospital director is deciding whether to perform a surgery to save a child’s life, he should not consider the surgery’s cost;^{Footnote 75} if a soldier is deciding whether to obey orders, she should not consider the benefits of disobedience.^{Footnote 76} These thought experiments about “taboo tradeoffs” or “incommensurate values” are so compelling that some philosophers consider the inability to trade off compliance with other goods to be the essence of moral norms.^{Footnote 77} This suggests that the motivation to comply with moral norms is not always of a common currency with other goods, and is not simply incorporated into people’s reward functions.

Two objections to this line of argument immediately come to mind. First, perhaps people place arbitrarily large negative reward on the outcome of certain norm violations, accounting for the apparent “unthinkable“ nature of noncompliance within a value-guided framework. For example, they might place infinitely negative reward on a state in which abortions are mandated, or organs are sold on markets. Thus, any benefit of violating the norm would be outweighed by the cost, and people would be unwilling to violate it. This interpretation appears to reconcile protected values with the reward theory of norm compliance, obviating the need for an alternative theory.

The consilience, however, is misleading. Rewards are common-currency, numerical representations of the subjective, intrinsic goodness in certain states or actions. An essential feature of such representations is that they can be traded off with each other.^{Footnote 78} Saying that a reward is too large to ever be outweighed is equivalent to saying that it is not a reward.

Second, even though people say they are unwilling to trade off compliance with other goods, perhaps they still would if the right situation arose. Perhaps their talk of taboos is just signaling (“I would never do that!”), or they are not imaginative enough to consider situations with truly enormous benefits of disobedience.^{Footnote 79} After all, when prompted, juries are able to put a price on something as sacred as a human life.^{Footnote 80} More work needs to be done to test whether, in their actual decisions, people are unwilling to put norm compliance on a common scale with other goods.

There is reason, however, to think that people genuinely treat compliance differently from other goods. People often go to absurd lengths to comply with trivial situational norms, far beyond what a reasonable cost-benefit, common-currency analysis would recommend. For instance, subjects in the Milgram experiment chose to fatally electrocute another human rather than disobey norms of proper conduct.^{Footnote 81} Do people value norms of proper conduct more than human life? More likely, the norms were influencing behavior through some non-value-guided channel, thus circumventing this comparison. What is this alternative channel?

E. Compliance by choice set construction: A model of “unthinkable“ action

Here, we suggest one possibility. The basic insight of the “taboo trade-offs“ literature is that people sometimes take norm violation off the table completely as an option. When people are unwilling to consider the benefits of organ markets, they are effectively eliminating selling their kidney from the set of choices they consider. On this view, moral norms aren’t just another factor in the expected value of an option; they influence which options even get their expected value calculated in the first place. They change which options get entertained (Figure 2).

This theory can be naturally formulated in the language of MDPs. A crucial variable in an MDP is the set of available actions A _s at each decision point s. This variable is subjective, and must be constructed by the decision-maker. It corresponds to what economists sometimes call the “choice set“ that a decision-maker constructs prior to choice, narrowing from all conceivable actions to a restricted set of considered alternatives. The idea is that norms influence choices in part by excluding norm-violating actions (for instance, selling your kidney) from A _s, for some broad set of decision points s. Thus, after being exposed to a norm, people become less likely to violate it.

As before, formulating the theory in the language of MDPs allows us to clarify some of its ambiguities. It would be foolish to claim that moral norms always eliminate norm-violating actions from A _s — people sometimes do sell their kidneys, and soldiers sometimes do disobey orders. The theory needs some kind of hedge to account for these exceptions. But where exactly should it hedge? One possibility is that the process that generates the action set A _s is probabilistic, and norms make it less likely (but not impossible) for norm-violating actions to make the cut. Alternatively, being exposed to a norm could cause you to deterministically exclude certain actions from A _s, but only in certain states s (for instance, never consider selling your kidney, unless you are in a state of desperation). MDPs can capture both possibilities.

There is also ambiguity in what it means to exclude an action from A _s. It could mean that the decision maker is aware of the action, but deems it irrelevant. This view is bolstered by the fact that people often deem norm-violating options irrelevant when judging others. For example, consider a captain on a sinking ship who throws either some cargo, or his wife, off the ship. In the latter case, people think the captain acted freely, because he could have thrown the cargo off instead. But in the former case, people think the captain was forced, because there was nothing else he could have done.^{Footnote 82} This belief persists even if people are reminded that he could have thrown his wife off the ship instead; they say that option is irrelevant. This tendency to deem norm-violating options irrelevant seems to appear whenever people judge others’ choices.^{Footnote 83} Perhaps people apply the same tendency to their own choices, and deem norm-violating options irrelevant to the decision-making process.

Alternatively, excluding an action from A _s could mean that the decision-maker is never aware of the action — it doesn’t even come to mind. (More precisely, the representation of a norm-violating action [“I should sell my kidney…“] is never tokened in the person’s mind.) While this seems unlikely in laboratory experiments with explicit, constrained choice sets (such as “cooperate“ or “defect“), real-world decisions typically have choice sets that are unbounded and unstructured (think of all the things you could conceivably do on a given Saturday afternoon). People never represent the vast majority of the choices they could make, and norms might influence which ones make the cut. The cognitive mechanisms behind choice set construction are extremely unclear, and so it is unclear how norms would produce this effect. But the idea is ripe for future research.

Why would a decision-maker ever exclude actions from consideration? Why not consider all of them? In laboratory experiments with constrained choice sets, it seems implausible that people wouldn’t evaluate every option. But again, in the real world, choice sets are enormous. People must have mechanisms for narrowing down their options to a small subset.^{Footnote 84} Since norm-violating options are typically unwise, people would obtain a practical benefit from excluding them.

Moreover, there is a strategic benefit to not even think about violating norms. If other people notice that you considered violating a norm (for example, you took two minutes before deciding to cooperate in a repeated economic game), they will infer that you would violate the norm if the incentives were strong enough, and thus terminate productive relationships with you. In other words, in repeated games where your partners know which options you consider, it is strategically beneficial to obey norms “without looking.“^{Footnote 85} These considerations further motivate the theory that norms cause people to ignore norm-violating options.

The possibility that norms exclude options from consideration stands out as an important new frontier for research. Some preliminary evidence comes from a study showing that, by default, people tend to treat immoral actions as if they were actually impossible.^{Footnote 86} Only after a little time and thought do they acknowledge that immoral actions are possible, albeit wrong. Although this pattern of response becomes most pronounced in adults subject to time pressure, it is promiscuous in the judgments of young children. For instance, at four years old, a majority of children claim that it is impossible to steal a candy bar from a store — and, moreover, that to succeed would require magic to do so.^{Footnote 87}