This is an extremely rich and cultured book that makes a large number of intelligent points about experimental methods. It also raises many sophisticated questions concerning what it means to test a theory and how one can test in an environment in which an error model unconstrained by theory is essential to judging empirical fit. It also looks to refute a number of in principle counterarguments against experimental economics, while at the same time taking a heavy pot shot at what it sees as the unnecessary domination of theory over fact in experimental economics. I agree with many of the individual points that are made in the book, but have a very different take on the appropriate relationship between economic theory and experimentation.
To summarize briefly, chapter 2 examines what it means to test a theory with experiments, and draws a useful distinction between the domain of behaviour that the theory is intended to characterize and the domain that the theory can be appropriately tested on. Chapter 3 points out how difficult it is to study strategic settings in the lab given the need simultaneously to infer subjects’ preferences, their understanding of the strategic environment, and their choice of strategy. Chapter 4 considers the manner in which experiments have evolved from being tasked with evaluating theories to generating regularities that demand theoretical attention. To some extent I agree with this point, although it is the implicit approval of going further in this direction that identifies my chief area of substantive disagreement. Chapter 6 concerns incentives, and is even-handed. While recognizing the value of the economists’ insistence on keeping the subject-pool free of deception, the authors point out that there is no clear definition of the restriction that this imposes in practice, and that it can limit scientific advance. Chapter 7 considers the topic of ‘noise’, and makes the important point that one cannot safely divorce the jobs of theorist, experimenter and econometrician. This is a deep point, and one that will require far more investigation over the coming years.
Chapter 5 takes on the issue of external validity, and the now-tense relationship between lab and field experiments. I feel that the appallingly weak anti-experimental case is not suitably ridiculed. Can anyone seriously believe that because field experiments involve real stakes, the experimental lab is somehow superfluous? There are just so many different fields out there! It is a fantasy to imagine that we, staring intently at our office walls, can identify what types of individual there are and what identifies a ‘field’ without directed research in the experimental lab. The fact that we have ever-deepening understanding of the typical experimental subject pool is in many ways positive. In addition, the experimental lab allows manipulations that are so much richer than those in the field. Instead of criticizing experimental economics a priori, it is time to move on and work on the hard business of how best to transition back and forth between lab and field.
So why, with so many areas of agreement, do I find myself somewhat confused by the book? Part of the answer is that I found it hard to identify a clear sense of direction, which may simply be the result of having many distinguished co-authors with subtly different points of view. Rather than coming across as a unified treatise, the book at times feels like a collection of interesting and important observations. There is no sense that the whole is larger than the sum of the parts. As a result, I found it hard to judge what purpose the authors intend the book to serve. It is certainly not comprehensive enough to serve as a textbook, and it is not detailed enough to serve as a technical guide for experimentalists.
While there is no explicit statement to this effect, the book does convey a strong sense of what the authors would most like to accomplish. I believe that they seek not only to guide up and coming researchers on how best to approach experimental economics as a scientific enterprise, but also to promote a particular vision for the future of the enterprise. In essence, they propose a major change in experimental methods that downgrades the role of economic theory and raises the role of experimentation as fact finder. Reading between the lines, they appear to believe that it is past time for economics to adopt the methods of psychology, and to allow experimental findings rather than theoretical reasoning, to be the lingua franca. As in psychology, the ideal they pose is to find low-level regularities and then refine them with ever more finely tuned experiments.
The discovery of loss aversion is the authors’ paradigm example of idealized experimental economics. For economic theorists, loss aversion, while not inconsistent with properly interpreted expected utility theory, was something of an afterthought. Yet experiment after experiment in humans and in animals suggests that loss aversion is close to universal. It is now seen as a rare example of a cross-species truth uncovered through experimentation. In the authors’ view, this is the ideal to which experimental economics should aspire: a switch from thinking through models to thinking through findings.
Given this outstanding success, the question that the authors pose is why economists have so long resisted following where experiments appear to lead. They see this as a reflection of an archaic methodology in which economists strive to identify through pure logic how people ‘should’ behave, resisting powerfully any evidence to the contrary. Rather than stick with this seemingly religious commitment, the authors argue that it is time for theorists to allow experimentalists to take the lead in guiding them on what forces to model. Theorists, in this interpretation, need to be good clean-up workers clarifying what the true (experimental) pioneers uncover.
I agree that we have various intuitions that are essentially pre‐theoretical, and that an important role of experiments is to explore these intuitions as a prelude to deep theoretical investigation. What I do not agree with is the apparent down‐grading of theory. In fact I see the basic division between a priori thinkers (theorists) and those searching for facts as false. What is more, by thinking in terms of such a dichotomy, I believe that the authors understate how profound and foundational are the contributions that experimental methods may make to economic theory in the upcoming period.
By way of background, I believe that good theorists neither treat their assumptions as self evident, nor look for general theories that have a rational flavour. They like logic and they like simplicity (who doesn't?). They are also aware of the traps involved in using conceptual labels that convey intuitive content, yet are hard to define, such as ‘loss aversion’ (even ‘loss’ and ‘aversion’ alone are hard to define). Those of us who are less impressed with the truths that psychological methods have uncovered worry a great deal about the ability to create a ‘false consensus’ through agreement on the centrality of concepts that we each define as we see fit. Precisely for reasons of linguistic clarity and communication, I believe that psychology will end up adopting methods of economics more than the other way around.
To see why I believe the division between theorists and experimentalists to be artificial, consider the theory of revealed preference. In this case, the theory actually suggested the idealized data (choice from arbitrary choice sets), which in turn suggested the value of experiments in which many choices are offered among sets of objects all selected from the same grand set, with significant overlap among available options in the various choice sets, precisely in order to give the theory empirical content. These revealed preference experiments have since become a mainstay of the profession, and the noted limitations of the theory are increasingly giving rise to ever more sophisticated theories that in turn demand ever more discriminating tests. In this co-evolutionary approach, theoretical constructs and the experimental designs are evolving at one and the same time.
How do these revealed preference experiments differ from experiments that test regret theory, which was pioneered and continues to be championed by (several of) the authors of the current book? Even more to the point, how do they differ from the tests of prospect theory, the paradigm case in the book itself? They differ in that revealed preference theory is fundamentally data theoretic. Samuelson's amazing twist on economic theory was to treat an idealized data set as synonymous with the theory of utility maximization. Rather than treating the intuitive theory of utility maximization as being primary, he treated the idealized data as primary, with the theory being identified with particular properties of this data set. The appropriate test of a Samuelsonian theory does not have to be inferred later: the test and the theory are tightly intertwined. In contrast, regret theory and the theory of loss aversion become animated in particular settings, but do not convey obvious general limitations on observed choice data.
What is so valuable about the case of revealed preference is that it presents a vision of a theory that the laboratory is absolutely necessary for testing (the authors’ cautionary notes in chapter 7 on how best to model errors is extremely well taken in this case). In recent work, my co-authors and I have applied revealed preference arguments to characterize various modes of search both in theory (cf. Caplin and Dean Reference Caplin and Dean2011) and in the corresponding experiments (Caplin et al. Reference Caplin, Dean and Martin2011). The theories have no implications whatever for standard choice data, leading us to consider data on the ‘choice process’, comprising provisional choices in the pre-decision period. The advantages of the experimental laboratory over the field stand out particularly starkly in research of this type. If one is to gather such ‘unnatural’ data to explore the fit of a theory, one has to design the interface with which the data are gathered, and then present it to subjects in a controlled setting. If those of us who mine this approach can identify rich enough seams, economic experimentation will grow ever more central in the social scientific enterprise.
