Suppose that, like Jason Bourne, you find yourself without memory. In an unfamiliar hotel room, two bodies at your feet, blood splattered on your clothes. In your right hand – a gun. In your left hand – a copy of Cushman's “Rationalization Is Rational.” What should you do now?
Cushman argues persuasively that it is entirely reasonable to go about taking actions in the world, in order to recover a policy from actions driven by non-beliefs and non-desires. You could leave your hotel room and start acting and reconstructing, armed with the knowledge that through evolution and habituation your actions probably make sense (also, armed with a gun). But you could also sit down on the hotel bed and have a think.
Thought experiments are part of representational exchange, according to Cushman, a way of learning about the world that is “beyond decision making.” There are different accounts of how learning from thought experiments works, with many suggesting an explicit unpacking of mental models that have implicit constraints (see e.g., Clement Reference Clement2009; Lombrozo Reference Lombrozo2017; Mach Reference Mach, Mach, McCormack and Foulkes1897/1976). However, thought experiments often do involve decision making, and the knowledge we gain through them is not necessarily about the world, but about the self. Here I have in mind thought experiments of the more everyday sort, the “would you rather” questions that people like to engage in, as opposed to “what would two blocks tied together to a string do when falling” that only very specific people like to engage in. But many moral reasoning problems fall under this category as well.
Cushman's framework can help explain why such everyday thought experiments are informative, and also why people like to engage in them. Assume that people do not have direct access to their own underlying reasons for action (whether beliefs, desires, habits, or something else) but rather construct a kind of belief-desire theory of themselves (e.g., Gopnik & Meltzoff Reference Gopnik, Meltzoff, Parker, Mitchell and Boccia1994; Saxe Reference Saxe, Markman, Klein and Suhr2009). A thought experiment that asks the thought experimenter what action they would take can engage non-rational (habitual, evolutionarily granted) decision-making modules to produce a hypothetical decision. This decision can in turn be used to update a person's theory of themselves, through a similar mechanism to the inversion of the reward from real actions for others (Baker et al. Reference Baker, Saxe and Tenenbaum2009; Reference Baker, Jara-Ettinger, Saxe and Tenenbaum2017). But all of this would be happening without setting foot out of the room. In the same way that Cushman posits an “offline planning” direction from planning to habit in representational exchange, this may be an “offline rationalization.”
People take pleasure in answering such thought experiments (McCoy et al. Reference McCoy, Paul, Ullman, Goldman and McLaughlin2019) because information gained in this way is rewarding, in the same way that any information gain or uncertainty reduction may be rewarding in and of itself (Auer et al. Reference Auer, Cesa-Bianchi and Fischer2002). This dynamic of answering from inaccessible modules and updating a theory of those modules can also explain how people can surprise themselves in such thought experiments (McCoy et al. Reference McCoy, Paul, Ullman, Goldman and McLaughlin2019), to the degree that there is a misalignment between the two.
So, Bourne could order some room service, pick up a book, and learn something about himself. However, the overall rational-rationalization account as inverse-policy-learning leaves out a possible central constraint that seems different for inverting one's own policy compared to inverting the policy of another: People are the heroes of their own story. When seeing unconscious cops at his feet, Bourne could reasonably conclude that he's a bad person. Anyone walking into the room at that moment would likely draw that conclusion, so why doesn't Bourne? Fanciful stories aside, there are many situations in which similar behavior driven by similar habits in ourselves and others are rationalized differently, in a way that is skewed in our favor. When I fail to study for a test, it is because the material is not engaging. When you fail to study, it is because you don't like to work hard. In reality, we were both just tired and hungry. Rationalization-is-rational can explain why people try to reconstruct mental variables in these situations as an adaptive behavior. But to the degree that it is adaptive through being often accurate, it seems odd that rationalization would often diverge in this systematic way – unless there was some additional difference to make a difference in this computation. And that difference would itself need to be explained on adaptive grounds.
This is a question about what, if anything, needs to go into the inversion of the policy to make rationalization different for myself and others. It is possible that this is a matter of different input information or missing information in that calculation: In addition to the action of not studying, I am also privy to certain mental states like the fact that I am not lazy. But this seems to be begging the question; the whole point of rationalization is that it reconstructs such states where there were none, without the awareness of the person doing the rationalization. An alternative is that there is an overarching, adaptive principle that ensures that rationalization for one's self is more in one's favor than inverse planning for others. This would be akin to one's own beliefs and desires being in line with what one sees as good or desirable. But if this comes at the expense of accurate inference, why have it at all?
In short, I am on board with Cushman's account of rationalization as adaptive, and as part of a broader account of representational exchange. If anything, I think this account can be broadened to capture the engaging aspect of thought experiments that involve making a decision. However, a full account of the functional role of rationalization will need to account not only for its self-benefitting nature, but also its self-serving one.
You have
Access
Suppose that, like Jason Bourne, you find yourself without memory. In an unfamiliar hotel room, two bodies at your feet, blood splattered on your clothes. In your right hand – a gun. In your left hand – a copy of Cushman's “Rationalization Is Rational.” What should you do now?
Cushman argues persuasively that it is entirely reasonable to go about taking actions in the world, in order to recover a policy from actions driven by non-beliefs and non-desires. You could leave your hotel room and start acting and reconstructing, armed with the knowledge that through evolution and habituation your actions probably make sense (also, armed with a gun). But you could also sit down on the hotel bed and have a think.
Thought experiments are part of representational exchange, according to Cushman, a way of learning about the world that is “beyond decision making.” There are different accounts of how learning from thought experiments works, with many suggesting an explicit unpacking of mental models that have implicit constraints (see e.g., Clement Reference Clement2009; Lombrozo Reference Lombrozo2017; Mach Reference Mach, Mach, McCormack and Foulkes1897/1976). However, thought experiments often do involve decision making, and the knowledge we gain through them is not necessarily about the world, but about the self. Here I have in mind thought experiments of the more everyday sort, the “would you rather” questions that people like to engage in, as opposed to “what would two blocks tied together to a string do when falling” that only very specific people like to engage in. But many moral reasoning problems fall under this category as well.
Cushman's framework can help explain why such everyday thought experiments are informative, and also why people like to engage in them. Assume that people do not have direct access to their own underlying reasons for action (whether beliefs, desires, habits, or something else) but rather construct a kind of belief-desire theory of themselves (e.g., Gopnik & Meltzoff Reference Gopnik, Meltzoff, Parker, Mitchell and Boccia1994; Saxe Reference Saxe, Markman, Klein and Suhr2009). A thought experiment that asks the thought experimenter what action they would take can engage non-rational (habitual, evolutionarily granted) decision-making modules to produce a hypothetical decision. This decision can in turn be used to update a person's theory of themselves, through a similar mechanism to the inversion of the reward from real actions for others (Baker et al. Reference Baker, Saxe and Tenenbaum2009; Reference Baker, Jara-Ettinger, Saxe and Tenenbaum2017). But all of this would be happening without setting foot out of the room. In the same way that Cushman posits an “offline planning” direction from planning to habit in representational exchange, this may be an “offline rationalization.”
People take pleasure in answering such thought experiments (McCoy et al. Reference McCoy, Paul, Ullman, Goldman and McLaughlin2019) because information gained in this way is rewarding, in the same way that any information gain or uncertainty reduction may be rewarding in and of itself (Auer et al. Reference Auer, Cesa-Bianchi and Fischer2002). This dynamic of answering from inaccessible modules and updating a theory of those modules can also explain how people can surprise themselves in such thought experiments (McCoy et al. Reference McCoy, Paul, Ullman, Goldman and McLaughlin2019), to the degree that there is a misalignment between the two.
So, Bourne could order some room service, pick up a book, and learn something about himself. However, the overall rational-rationalization account as inverse-policy-learning leaves out a possible central constraint that seems different for inverting one's own policy compared to inverting the policy of another: People are the heroes of their own story. When seeing unconscious cops at his feet, Bourne could reasonably conclude that he's a bad person. Anyone walking into the room at that moment would likely draw that conclusion, so why doesn't Bourne? Fanciful stories aside, there are many situations in which similar behavior driven by similar habits in ourselves and others are rationalized differently, in a way that is skewed in our favor. When I fail to study for a test, it is because the material is not engaging. When you fail to study, it is because you don't like to work hard. In reality, we were both just tired and hungry. Rationalization-is-rational can explain why people try to reconstruct mental variables in these situations as an adaptive behavior. But to the degree that it is adaptive through being often accurate, it seems odd that rationalization would often diverge in this systematic way – unless there was some additional difference to make a difference in this computation. And that difference would itself need to be explained on adaptive grounds.
This is a question about what, if anything, needs to go into the inversion of the policy to make rationalization different for myself and others. It is possible that this is a matter of different input information or missing information in that calculation: In addition to the action of not studying, I am also privy to certain mental states like the fact that I am not lazy. But this seems to be begging the question; the whole point of rationalization is that it reconstructs such states where there were none, without the awareness of the person doing the rationalization. An alternative is that there is an overarching, adaptive principle that ensures that rationalization for one's self is more in one's favor than inverse planning for others. This would be akin to one's own beliefs and desires being in line with what one sees as good or desirable. But if this comes at the expense of accurate inference, why have it at all?
In short, I am on board with Cushman's account of rationalization as adaptive, and as part of a broader account of representational exchange. If anything, I think this account can be broadened to capture the engaging aspect of thought experiments that involve making a decision. However, a full account of the functional role of rationalization will need to account not only for its self-benefitting nature, but also its self-serving one.