WHAT IS REPLICATION?

Barba (Reference Barba2018) points out that although there has been prolific discussion of the terms “reproducibility” and “replication” in many disciplines in recent years, confusion and conflicting uses are widespread. In her survey of relevant literature, Barba finds that some fields make no distinction between “reproducibility” and “replication.” Among fields that do recognize a distinction between the two, the meanings are sometimes directly inverted. Here we follow what Barba has identified as the most common, long-established, and highly cited definitions of these terms, as also recently recommended by the National Academies of Sciences, Engineering, and Medicine (2019). Reproducibility is the ability to obtain results by using the same data, code, and procedures provided by the original authors (Marwick Reference Marwick2017). This is only possible when the authors make all those materials available; for example, in a research compendium (Marwick et al. Reference Marwick, Boettiger and Mullen2018). Replication is the ability to arrive at the same scientific conclusions in a new study, collecting new data (possibly with different methods) and completing new analyses.

In the following section, we briefly survey replication assignments described in other fields to show the variety of forms this usually takes. Replication assignments across different fields may not fit strictly into the above definition of replication because they do not always involve a completely new study. Nevertheless, we consider that if a study departs from any of the original materials (e.g., new data with previously published code, or new code with previously published data), then it fits broadly within the definition of replication.

HOW DO DIFFERENT DISCIPLINES USE REPLICATION ASSIGNMENTS?

Some of the earliest discussions of replication in university curricula appear in economics and psychology (Höffler Reference Höffler2013; Standing et al. Reference Standing, Grenier, Lane, Roberts and Sykes2014). Ball and Medeiros (Reference Ball and Medeiros2012) describe their TIER (Teaching Integrity in Empirical Research) protocol for undergraduate economics students at Haverford College. This is intended to ensure that a student's work is replicable by the instructors. When students submit their final project report, their submission must contain four elements: the raw data files, a metadata file, script files of code used to analyze the data, and public availability (i.e., deposit in an open repository such as Dataverse). Frank and Saxe (Reference Frank and Saxe2012) describe how they teach undergraduates (at MIT) and graduate students (at Stanford) to do in-class replications of recent, cutting-edge psychology experiments, and note that several projects from their undergraduate course have even been part of successful publications. More recently Hawkins and authors (Reference Hawkins, Smith, Au, Arias, Catapano, Hermann, Keil, Lampinen, Raposo, Reynolds, Salehi, Salloum, Tan and Frank2018) reported on 11 replication assignments from a psychology graduate seminar at Stanford, finding that the replications typically yielded effects that were smaller than the originally published ones. Similarly, Jern (Reference Jern2018) describes how students in a psychology course completed replication assignments by using statistical methods of the original research articles with new data collected by the students outside of class.

Students in Stanford University's graduate course Advanced Topics in Networking are given a replication assignment in which they are asked to replicate “classic” computer networking experiments (Yan and McKeown Reference Yan and McKeown2017). Students work in pairs and receive modest instructor support. The assignment entails selecting appropriate emulation software, communicating with the original authors, obtaining the authors’ materials, replicating the experiment, and publicizing their results through both an in-class presentation and a blog post on a program website. We classify this as a replication assignment because many students could not obtain the original code from the authors and had to write their own for the assignment. Since 2012, more than 200 undergraduate and graduate students have participated in this assignment with an 86% success rate (Yan and McKeown Reference Yan and McKeown2017). Student feedback suggests high satisfaction with the assignment, citing unique educational value, improved understanding of the original material, and the acquisition of professional skills. In some cases, students personally contributed to the network engineering literature when their replications exposed inaccuracies in original experiments, which were then presented to and publicly amended by the authors (Yan and McKeown Reference Yan and McKeown2017).

In describing her political science classes at the University of Cambridge, Janz (Reference Janz2016) argues that reproducibility and replication should be held as the gold standard for scientific research. She states that teaching these concepts should be a necessary component of graduate studies, to ensure students can make their own future work reproducible. Janz reports on her class in which about 15 students undertake replication assignments over eight weeks, including providing weekly updates to each other to gain insight and feedback. Janz describes two possible levels of assignment suitable for different lengths of the term and levels of the students: duplication (aiming for the exact same results based on the exact same dataset with exactly the same methods) and replication (testing the robustness of previous research results by employing newly collected data, new variables, or new model specifications). Duplication, which we would define here as reproduction, may be beneficial for lower-level students, while upper-level students can replicate a study and contribute original data, potentially leading to publication. Janz (Reference Janz2016) describes how replication assignments are a growing trend in political science departments (noting that R and STATA are commonly used) and reviews many of the practical challenges of doing replication assignments in a graduate course. She also responds to six typical criticisms of replication assignments and points out the need for universities to nurture a culture of reproducibility and replication to ensure that the gold standard of reliable, credible, and valid research is not just an empty phrase.

The Freie Universität Berlin extracurricular graduate seminar course Digital Open Science aims to teach open science practice and assigns replication projects, mostly involving neuroimaging topics (Toelch and Ostwald Reference Toelch and Ostwald2018). These projects are carried out with a variety of typical open science software tools and services, including Python, R, Git, GitHub, and the Open Science Framework. Students first receive extensive lectures and hands-on tutoring, then choose a simple neuroscience experiment to replicate. Finally, they present their results at a symposium. The course's primary goal is to teach students the value of verifying data on which their own future research might rely. Students have reported a high rate of engaging in open science practices after taking the class, and 80% of the participants said that they believed the open science techniques would improve their future research as professionals (Toelch and Ostwald Reference Toelch and Ostwald2018). Millman and authors (Reference Millman, Brett, Barnowski and Poline2018) describe a similar course at the University of California at Berkeley that teaches students how to use open science tools to complete a capstone replication assignment on neuroscience topics.

This brief survey demonstrates that replication assignments are widely known in economics, psychology, political science, neuroscience, and other fields (e.g., Roettger and Baer-Henney Reference Roettger and Baer-Henney2018). Common elements include group work, use of open source software and services to make the replication results openly accessible to anyone, and a scaffolded, stepwise approach to the task to ensure that students receive instructor support at multiple stages throughout the assignment. To the best of our knowledge, replication assignments are not common in archaeology programs, although the tools and data structures are generally similar to other social sciences. We posted a message to the Society for American Archaeology Teaching Archaeology Interest Group e-community on May 27, 2019, to ask for examples of replication assignments used in teaching archaeology and received no replies from anyone teaching with replication. More broadly in archaeology, replication and reproducibility have received limited, but growing, attention. Elsewhere, we have documented recent rapid increases in the number of publications that include code and data to enable readers to reproduce the published results (Schmidt and Marwick Reference Schmidt and Marwick2019).

HOW TO CONDUCT A REPLICATION ASSIGNMENT IN ARCHAEOLOGY

In this section, we describe our replication assignment and how we assessed its effectiveness. A brief discussion of our replication report assignment was announced at the beginning of the 10-week-long term to give students background about the purpose and concepts of replication and our expectations. Our replication report assignment consisted of three small, graded activities to scaffold the preparation of the final report. The first step started from Week Four and each step was separated by one week to give students time to work and submit their final reports, due on Week Seven. Students were expected to work in groups of three to four people but to submit their assignments for each of the three steps and the final report individually. Submissions for each step were graded as complete/incomplete, with feedback provided individually via the Canvas learning management system, and collectively during class meetings. Our course had no prerequisites, so we assumed no prior knowledge of the free and open source R programming language among the students and were prepared to teach them as complete novices. We chose R (R Core Team 2019) because it is widely used by archaeologists (Schmidt and Marwick Reference Schmidt and Marwick2019), and also commonly taught in undergraduate classes in social sciences and statistics (Baumer et al. Reference Baumer, Cetinkaya-Rundel, Bray, Loi and Horton2014; Cetinkaya-Rundel and Rundel Reference Çetinkaya-Rundel and Rundel2018; Dvorak et al. Reference Dvorak, Halliday, O'Hara and Swoboda2019). We were also prepared for students to have no prior experience with replication assignments.

OBSERVATIONS ON THE ASSIGNMENT PROCESS

The first step, choosing the target article, revealed the need for some intervention from the instructor to guide students to articles that used relatively simple statistical methods. For example, one group initially chose Breslawski and authors (Reference Breslawski, Etter, Jorgeson and Boulanger2018) as their target article, but the key claims in this paper depend on multiple comparisons of multilevel regression models. We explained to the students that if they attempted to replicate a key claim of this paper, then they would likely be doing substantially more work than other groups in the class. We invited this group to choose a different target article to ensure a more comparable experience, which they accepted. The statistical backgrounds of our students were highly diverse, so we could not expect students to be very discerning about the statistical complexity of the methods used in the potential target articles. As a consequence, we were prepared to intervene to guide their selection of a paper that we could be sure they could successfully replicate, given the time available. The target articles used by this class were Marwick and authors (Reference Marwick, Clarkson, O'Connor and Collins2016), Bicho and Cascalheira (Reference Bicho, Cascalheira, Picin and Cascalheira2020), and Marwick (Reference Marwick2013).

The second step was mostly straightforward, with students engaging in discussion in class and on Slack to identify the two to three key claims of their target paper as well as identifying the data visualization that provided the most relevant support to one or more of those claims. Given the varied statistical background knowledge of the students, during lectures we covered some statistical methods they might encounter, such as principal components analysis, to give them the mathematical concepts behind them. Identifying the data files was less straightforward, with about one-third of students failing to correctly identify the data files accompanying their target article. We attribute this to the relatively low level of familiarity of the students in working with raw data such that they were not sure when they were looking at it, and to the high degree of variability in how the target article authors made their data open. Some authors included their data as a file in the supplementary information attached to the article, while others deposited their files in an open data repository such as osf.io or figshare.com, and then cited the DOI to the files in their article. When the data files were nested in several layers of folders, some students struggled to find them.

The ability to easily share screenshots on Slack was important to the success of the third step. Our intention was that two lab classes earlier in the term that introduced students to some methods for data visualization using R would provide the foundation for succeeding in this step. We expected that two lab reports completed earlier in the term that were required to be written with R Markdown would help students practice crucial code they might need later. For the lab reports, students used R Markdown templates that we provided to complete tasks of reading data into R, basic data tidying, and visualizing data by modifying sample code. However, we found that for some students this was not sufficient practice, and substantial instructor guidance was required to help them complete this step. We then met one-on-one with each group to check how successfully they had produced a basic visualization using data from the target article, and to discuss the group's strategy to complete the report. This was the most time-consuming aspect of the assignment for the instructor, involving a one-hour meeting with each of the five groups.

ANALYSIS OF THE STUDENTS’ ANONYMOUS FEEDBACK

Thirteen out of 16 students completed an anonymous feedback survey (Figure 1). Only one student had done replication before, and two had used R previously for an archaeology assignment. Most students strongly agreed with the statements about having sufficient support and clear instructions. Most students strongly disagreed with the statement “I am likely to attempt to replicate published research in my future studies and work.” This contrasts with the high proportion of students who agreed with the statement “The ability to replicate published research is an important skill for professional archaeologists.” Taken together, these two responses show that while students see the value of replication for archaeology in general, they do not see any specific benefits to doing it themselves. This may reflect a failure of the instructor to communicate the individual benefits of developing skills for replication. It may also reflect uncertainty among the students about their plans for a career in archaeology. Most, but not all, students agreed that the replication assignment helped them hone their research skills more effectively than reading a paper would have helped them learn to write a traditional paper.

FIGURE 1. Results of the anonymous feedback survey on the replication assignment.

Figure 2 shows the correlations between the five feedback questions that have Likert scale responses. The statements about instructions and instructor support are highly positively correlated, showing the positive effects of the assignment design, a detailed rubric, and the instructor meeting with each group to discuss their work and answering students’ questions promptly on Slack. The strongest negative correlation is between the statements about instructor support and doing replication in future work. This might suggest that the students received so much support that they did not feel capable of doing a replication like this by themselves. We see confirmation of this in the free-form comments, such as “it would not have been possible for us to do this correctly on our own.” These correlations indicate a need to equip students with skills to work more independently of the instructor and to strengthen students’ self-efficacy with replication skills.

FIGURE 2. Correlations among feedback items with Likert scale responses. The size of the dot indicates the magnitude of the correlation, and the color indicates the direction (red is negative, blue is positive). Correlations were computed using Spearman's (Reference Spearman1904) method.

ANALYSIS OF THE STUDENTS’ GRADES FOR THE REPLICATION ASSIGNMENT

We graded the students’ final submissions using a rubric with criteria that covered content; the introduction, methods, result, and conclusion sections; and style. In Figure 3 we show the distribution and means of student scores for each criterion. The two criteria showing the highest mean score are “Style: use commas and apostrophes correctly, and spell consistently,” and “Intro: has clear statement of the purpose of the report.” High scores for the grammar criterion are expected because these reflect basic writing skills required for many undergraduate-level courses. Students are expected to have learned these in lower-level classes before taking this class. The high scores relating to the introduction section may reflect the effectiveness of the scaffolding steps that helped students focus on the specific purpose of the assignment. The lowest mean score is for “Content: minimum of 4 scholarly items in the reference list,” which shows that some people did not include four items. This might result from insufficient prior training in searching for scholarly publications, suggesting that although this is also a skill that should have been acquired before taking this class, many students remain weak at this task. A low mean is also evident for “Intro: has names, locations, and basic chronology of sites,” because some students neglected to supply these archaeological details. Future use of this assignment will incorporate these low-scoring criteria into the scaffolding steps to emphasize their importance to students and provide an opportunity for early feedback.

FIGURE 3. Distribution of students’ scores across the grading rubric criteria. Each point is one student. Red lines indicate the mean score for all students per criterion.

The criteria most relevant to the replication component of the assignment, in order from highest mean score to lowest, are “Content: submission includes Rmd file, Data file, and Word file”; “Conclusion: state whether the author's claims appear to be robust, unreliable, etc.”; “Results: includes 1-2 original plots & description of these”; and “Methods: identify the specific results you will replicate.” This suggests that we could help students develop better skills in narrating their process (writing about methods) and in describing and interpreting their data visualizations. In the future we may include more fundamental exercises focusing on these tasks in the scaffolding steps. Overall, we find that comparison of the scores for the replication criteria and other criteria shows there is no clear evidence that the replication component of this assignment lowers students’ grades. The two lowest scoring criteria are more generic research and writing skills rather than skills specific to the replication aspect of the assignment.