Many scientists, including the authors of the target article, use the metaphor of “building a wall” for accumulating scientific knowledge. In this analogy, correcting a false positive through replication is compared to removing a faulty brick from the wall. This leads to the widespread view that conducting replications only has the potential to eliminate knowledge and not to add to knowledge. Here we contend that this view is not only misguided, but also detrimental to the field.
In their target article, Zwaan et al. note that a common concern about direct replications is that they have little theoretical value (concern II, sect. 5.2). For example, according to Stroebe and Strack (Reference Stroebe and Strack2014, p. 63), “one reason why exact replications are not very interesting is that they contribute little to scientific knowledge.” Their account portrays original research as a constructive process – the laying of the bricks – and replication as a secondary process – the testing of the bricks. Replication research, which can only tear down weak parts of the wall (or confirm that existing bricks are strong), is considered of lower theoretical and scientific value than original research, which can help build new parts of the wall.
There are some senses in which this analogy is quite apt. For example, because conducting an original study requires design, it often involves more work (and more creativity) than does running a replication. The original study is also generative in the sense that it posits the idea, whereas the replication tests an existing idea. For these reasons, our field's emphasis on original research doesn't seem entirely misplaced. One skill we should value among scientists is the ability to come up with novel ideas, or with novel ways of testing existing ideas.
There are other aspects of this analogy, however, that we see as more problematic. First, it creates a false distinction between original and replication studies, treating the first study as a greater contribution to knowledge than those that follow. Science does not care whether data come from a replication or an original study. The order in which these explorations take place is orthogonal to the degree to which they advance our understanding. Gelman (Reference Gelman2016b) illustrates the problem with this way of thinking using the “time-reversal heuristic” in which a reader is asked to imagine that the replication study was done first. This exercise is meant to show that the order in which studies are conducted should be irrelevant to our evaluation of their scientific value.
A further weakness of the “building a brick wall” analogy is that equating an original study to a new brick suggests that researchers are in the business of inventing psychological phenomena. In reality, we are simply trying to understand the psychological phenomena that already exist. If we think of effects as existing in the world, prior to any particular scientific investigation of them, we realize that the distinction between original research and replication research is arbitrary from the point of view of quantifying the evidence provided.
In addition to being inaccurate, it may also be harmful to think of replication research as fundamentally different from original research. When original studies are presented as constructive and knowledge-producing (i.e., “laying the bricks”), whereas replication studies are presented as a mechanical, auditing activity (i.e., “testing the bricks”), this reduces the perceived value of, and therefore incentive to conduct, replication research (Crandall & Sherman Reference Crandall and Sherman2016). This is problematic given that replication is one of the defining features of a scientific field. Moreover, giving privileged status to original research contributes to the persistence of false-positive original findings in the literature, in textbooks, and in the media. When the scientific evidence produced by replication studies is perceived as different from (and often lesser than) the evidence produced by original studies, false claims from original studies become even harder to correct.
With these ideas in mind, we propose a new metaphor for scientific progress. Rather than likening scientific progress to building a wall, we suggest the analogy of solving a jigsaw puzzle. First, this highlights the fact that we are not builders but discoverers; we are not creating phenomena, but instead trying to reveal a pre-existing reality. Second, it highlights the nonlinear nature of progress; realizing that a piece is in the wrong spot is just as valuable as putting it in the right spot.
With this metaphor, it becomes apparent that original research and replication research are not as different as we might think. The order in which studies are carried out is not important for their evidentiary value; what is important is whether the result improves our understanding of reality (i.e., whether the puzzle piece is in the right place). Moreover, the misguided ideas of “constructive” and “destructive” research are avoided when thinking about solving a jigsaw puzzle. Any new evidence that moves us closer to an accurate solution is a constructive step in the process.
Many scientists, including the authors of the target article, use the metaphor of “building a wall” for accumulating scientific knowledge. In this analogy, correcting a false positive through replication is compared to removing a faulty brick from the wall. This leads to the widespread view that conducting replications only has the potential to eliminate knowledge and not to add to knowledge. Here we contend that this view is not only misguided, but also detrimental to the field.
In their target article, Zwaan et al. note that a common concern about direct replications is that they have little theoretical value (concern II, sect. 5.2). For example, according to Stroebe and Strack (Reference Stroebe and Strack2014, p. 63), “one reason why exact replications are not very interesting is that they contribute little to scientific knowledge.” Their account portrays original research as a constructive process – the laying of the bricks – and replication as a secondary process – the testing of the bricks. Replication research, which can only tear down weak parts of the wall (or confirm that existing bricks are strong), is considered of lower theoretical and scientific value than original research, which can help build new parts of the wall.
There are some senses in which this analogy is quite apt. For example, because conducting an original study requires design, it often involves more work (and more creativity) than does running a replication. The original study is also generative in the sense that it posits the idea, whereas the replication tests an existing idea. For these reasons, our field's emphasis on original research doesn't seem entirely misplaced. One skill we should value among scientists is the ability to come up with novel ideas, or with novel ways of testing existing ideas.
There are other aspects of this analogy, however, that we see as more problematic. First, it creates a false distinction between original and replication studies, treating the first study as a greater contribution to knowledge than those that follow. Science does not care whether data come from a replication or an original study. The order in which these explorations take place is orthogonal to the degree to which they advance our understanding. Gelman (Reference Gelman2016b) illustrates the problem with this way of thinking using the “time-reversal heuristic” in which a reader is asked to imagine that the replication study was done first. This exercise is meant to show that the order in which studies are conducted should be irrelevant to our evaluation of their scientific value.
A further weakness of the “building a brick wall” analogy is that equating an original study to a new brick suggests that researchers are in the business of inventing psychological phenomena. In reality, we are simply trying to understand the psychological phenomena that already exist. If we think of effects as existing in the world, prior to any particular scientific investigation of them, we realize that the distinction between original research and replication research is arbitrary from the point of view of quantifying the evidence provided.
In addition to being inaccurate, it may also be harmful to think of replication research as fundamentally different from original research. When original studies are presented as constructive and knowledge-producing (i.e., “laying the bricks”), whereas replication studies are presented as a mechanical, auditing activity (i.e., “testing the bricks”), this reduces the perceived value of, and therefore incentive to conduct, replication research (Crandall & Sherman Reference Crandall and Sherman2016). This is problematic given that replication is one of the defining features of a scientific field. Moreover, giving privileged status to original research contributes to the persistence of false-positive original findings in the literature, in textbooks, and in the media. When the scientific evidence produced by replication studies is perceived as different from (and often lesser than) the evidence produced by original studies, false claims from original studies become even harder to correct.
With these ideas in mind, we propose a new metaphor for scientific progress. Rather than likening scientific progress to building a wall, we suggest the analogy of solving a jigsaw puzzle. First, this highlights the fact that we are not builders but discoverers; we are not creating phenomena, but instead trying to reveal a pre-existing reality. Second, it highlights the nonlinear nature of progress; realizing that a piece is in the wrong spot is just as valuable as putting it in the right spot.
With this metaphor, it becomes apparent that original research and replication research are not as different as we might think. The order in which studies are carried out is not important for their evidentiary value; what is important is whether the result improves our understanding of reality (i.e., whether the puzzle piece is in the right place). Moreover, the misguided ideas of “constructive” and “destructive” research are avoided when thinking about solving a jigsaw puzzle. Any new evidence that moves us closer to an accurate solution is a constructive step in the process.