Hostname: page-component-745bb68f8f-b95js Total loading time: 0 Render date: 2025-02-11T15:11:04.188Z Has data issue: false hasContentIssue false

Scientific progress is like doing a puzzle, not building a wall

Published online by Cambridge University Press:  27 July 2018

Alexa M. Tullett
Affiliation:
Department of Psychology, University of Alabama, Tuscaloosa, AL 35487-0348. alexa.tullett@gmail.comhttp://alexatullett.com
Simine Vazire
Affiliation:
Department of Psychology, University of California, Davis, California 95616. svazire@ucdavis.eduhttp://simine.com

Abstract

We contest the “building a wall” analogy of scientific progress. We argue that this analogy unfairly privileges original research (which is perceived as laying bricks and, therefore, constructive) over replication research (which is perceived as testing and removing bricks and, therefore, destructive). We propose an alternative analogy for scientific progress: solving a jigsaw puzzle.

Type
Open Peer Commentary
Copyright
Copyright © Cambridge University Press 2018 

Many scientists, including the authors of the target article, use the metaphor of “building a wall” for accumulating scientific knowledge. In this analogy, correcting a false positive through replication is compared to removing a faulty brick from the wall. This leads to the widespread view that conducting replications only has the potential to eliminate knowledge and not to add to knowledge. Here we contend that this view is not only misguided, but also detrimental to the field.

In their target article, Zwaan et al. note that a common concern about direct replications is that they have little theoretical value (concern II, sect. 5.2). For example, according to Stroebe and Strack (Reference Stroebe and Strack2014, p. 63), “one reason why exact replications are not very interesting is that they contribute little to scientific knowledge.” Their account portrays original research as a constructive process – the laying of the bricks – and replication as a secondary process – the testing of the bricks. Replication research, which can only tear down weak parts of the wall (or confirm that existing bricks are strong), is considered of lower theoretical and scientific value than original research, which can help build new parts of the wall.

There are some senses in which this analogy is quite apt. For example, because conducting an original study requires design, it often involves more work (and more creativity) than does running a replication. The original study is also generative in the sense that it posits the idea, whereas the replication tests an existing idea. For these reasons, our field's emphasis on original research doesn't seem entirely misplaced. One skill we should value among scientists is the ability to come up with novel ideas, or with novel ways of testing existing ideas.

There are other aspects of this analogy, however, that we see as more problematic. First, it creates a false distinction between original and replication studies, treating the first study as a greater contribution to knowledge than those that follow. Science does not care whether data come from a replication or an original study. The order in which these explorations take place is orthogonal to the degree to which they advance our understanding. Gelman (Reference Gelman2016b) illustrates the problem with this way of thinking using the “time-reversal heuristic” in which a reader is asked to imagine that the replication study was done first. This exercise is meant to show that the order in which studies are conducted should be irrelevant to our evaluation of their scientific value.

A further weakness of the “building a brick wall” analogy is that equating an original study to a new brick suggests that researchers are in the business of inventing psychological phenomena. In reality, we are simply trying to understand the psychological phenomena that already exist. If we think of effects as existing in the world, prior to any particular scientific investigation of them, we realize that the distinction between original research and replication research is arbitrary from the point of view of quantifying the evidence provided.

In addition to being inaccurate, it may also be harmful to think of replication research as fundamentally different from original research. When original studies are presented as constructive and knowledge-producing (i.e., “laying the bricks”), whereas replication studies are presented as a mechanical, auditing activity (i.e., “testing the bricks”), this reduces the perceived value of, and therefore incentive to conduct, replication research (Crandall & Sherman Reference Crandall and Sherman2016). This is problematic given that replication is one of the defining features of a scientific field. Moreover, giving privileged status to original research contributes to the persistence of false-positive original findings in the literature, in textbooks, and in the media. When the scientific evidence produced by replication studies is perceived as different from (and often lesser than) the evidence produced by original studies, false claims from original studies become even harder to correct.

With these ideas in mind, we propose a new metaphor for scientific progress. Rather than likening scientific progress to building a wall, we suggest the analogy of solving a jigsaw puzzle. First, this highlights the fact that we are not builders but discoverers; we are not creating phenomena, but instead trying to reveal a pre-existing reality. Second, it highlights the nonlinear nature of progress; realizing that a piece is in the wrong spot is just as valuable as putting it in the right spot.

With this metaphor, it becomes apparent that original research and replication research are not as different as we might think. The order in which studies are carried out is not important for their evidentiary value; what is important is whether the result improves our understanding of reality (i.e., whether the puzzle piece is in the right place). Moreover, the misguided ideas of “constructive” and “destructive” research are avoided when thinking about solving a jigsaw puzzle. Any new evidence that moves us closer to an accurate solution is a constructive step in the process.

References

Crandall, C. S. & Sherman, J. W. (2016) On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology 66:9399. Available at: http://doi.org/10.1016/j.jesp.2015.10.002.Google Scholar
Gelman, A. (2016b) The time-reversal heuristic – A new way to think about a published finding that is followed up by a large, preregistered replication (in context of Amy Cuddy's claims about power pose). Blog post. Available at: http://andrewgelman.com/2016/01/26/more-power-posing/.Google Scholar
Stroebe, W. & Strack, F. (2014) The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science 9:5971. Available at: http://doi.org/10.1177/1745691613514450.Google Scholar