Incremental replications

We might think of replication as a practice that occurs in a separate lab with different researchers. Also the other two replication practices I will discuss can occur that way. But our own studies can also benefit from the verification of replication, within our own lab and within the same studies. An obvious step is to conduct exact replications within a series of experiments (e.g., “Experiment 2: Replication. We tested an additional 120 subjects using the same materials and procedures as we used in Experiment 1,” and “Experiment 4: Replication. We tested an additional 120 subjects using the same materials and procedures as we used in [Experiment 3],” Gernsbacher & Hargreaves Reference Gernsbacher and Hargreaves1988, p. 704 and 706).

More parsimoniously, we can conduct, within the same study, what I am calling incremental replications. For example, in a series of experiments investigating how readers understand pronouns, I probed participants immediately before versus after they read a pronoun in one experiment. In another experiment, I again probed participants immediately after they read a pronoun, but in this second experiment I also probed them after they finished reading the entire sentence (Gernsbacher Reference Gernsbacher1989). In this way, across experiments but within the same study, I tried to incrementally replicate each of the previously tested probe points (see also Garnham et al. Reference Garnham, Traxler, Oakhill and Gernsbacher1996, for a similar approach).

As another example, in a series of priming experiments, we manipulated two types of primes in a first experiment and manipulated again one of those two prime types along with a different prime type in a second experiment (Gernsbacher et al. Reference Gernsbacher, Keysar, Robertson and Werner2001a). In another series, we manipulated three prime types in a first experiment and repeated two of the three prime types across other experiments (Gernsbacher et al. Reference Gernsbacher, Robertson, Werner and Gorfein2001b). These incremental replications in within-subject designs also allowed us to assess the stability of our previous results in slightly different contexts (the value of which Zwaan et al. highlight).

Incremental replication is also valuable in between-subject designs. For example, in a series of between-subject treatment experiments, we repeated the baseline condition in subsequent experiments with other subjects (and juxtaposed with other treatments), which allowed us to assess the baseline condition's stability (Traxler & Gernsbacher Reference Traxler and Gernsbacher1992). In another series of experiments, we repeated the control condition in subsequent experiments, which allowed us to assess its stability (Traxler & Gernsbacher Reference Traxler and Gernsbacher1993).

Reciprocal replications

We might also think of replication as a practice that occurs only after a study has been peer reviewed. However, I would rather receive confirmation (or disconfirmation) of the stability of my results earlier rather than later. Zwaan (Reference Zwaan2017) in material left on the target article's editing floor, describes how this can be done.

Zwaan (Reference Zwaan2017) calls this practice as concurrent replication, and my recommendation goes one step further: Make the process reciprocal. Lab A attempts to replicate Lab B's study, while Lab B is doing the same for Lab A's study. Platforms such as StudySwap (deemed “a Craigslist for researchers” by Nosek in Chawla Reference Chawla2017) and Psychological Science Accelerator are ideal for reciprocal replication. Reciprocal replications should take some of the adversarial sting out of traditional replications.

Didactic replications

Lastly, we can make replication more mainstream by embracing it as a training tool. When I was a first-year doctoral student, in one of my first meetings with my advisor, he walked to his filing cabinet, pulled out a recently published article, and suggested I spend my first semester trying to replicate the results. The fact that this didactic activity occurred nearly 40 years ago might be surprising. More surprising might be the fact that the first author of the study my advisor tasked me to replicate was, indeed, my advisor (Foss & Blank Reference Foss and Blank1980).

As it turned out, the previous study only partially replicated (Foss & Gernsbacher Reference Foss and Gernsbacher1983). Learning how to execute an experiment from a published article was an incredibly valuable training experience. (Most likely this is why beginning cooks are encouraged to follow a recipe precisely, before adding their own flourishes.) Deciphering why the previous study only partially replicated was an even more valuable training experience. I believe I learned more about experimental design, stimulus creation, and the myriad other steps involved in doing good science than I would have learned had I joined an in-process study or tried to generate a new study from scratch.

The didactic value of replication has been advocated by others, most notably Grahe and his “Collaborative Replications and Education Project” (Grahe et al. Reference Grahe, Brandt, IJzerman and Cohoon2014). Along with Incremental Replications, which are replications built into each of a series of experiments to attempt to replicate parts of previous experiments, and Reciprocal Replications, which are reciprocal arrangements of co-replication, Didactic Replications can make replication more mainstream.