The last quarter-century has witnessed an ascendance of experimental methods in political science (Druckman and Lupia Reference Druckman and Lupia2012; McDermott Reference McDermott2002; Morton and Williams Reference Morton and Williams2010; Rogowski Reference Rogowski and Scott2016). One reflection of this development has been an ostensible growth in the number of departments that have experimental laboratories. Labs, and the subject pools that often accompany them, entail not only nontrivial resource investments but also can create a “research hub” that shapes a department’s intellectual pursuits. One consequence is that graduate students interested in lab-experimental research may be more likely to enroll in programs with labs—and even those who are not initially interested may develop such a focus. The emergence of lab experiments has been paralleled by the evolution of survey experiments implemented with nationally representative samples and convenience samples of respondents drawn from online platforms (Berinsky, Huber, and Lenz Reference Berinsky, Huber and Lenz2012; Krupnikov and Levine Reference Krupnikov and Levine2014; Mullinix et al. Reference Mullinix, Leeper, Druckman and Freese2015; Sniderman Reference Sniderman, Druckman, Green, Kuklinski and Lupia2011). Whether conducted in a lab or in a web-based survey, experiments create a challenge and an opportunity: given finite time and resources, how can we ensure that interested scholars have access to participants and how can we best take advantage of the collectivity that forms around research groups composed of individuals engaged in similar methods?
This article discusses our experiences in building institutionalized procedures so as to facilitate access and channel collective energy. The central aim of these efforts is to enhance the quality of graduate advising. This is particularly important given that the number of graduate students involved in experimental research may exceed the number of faculty (at least in our case) and institutions can ensure that graduate students have access to feedback and data.
A few caveats are in order. First, ours is only one experimental research group at a PhD-granting institution (i.e., Northwestern University), and most of those involved take a quantitative approach to studying political behavior in the United States.Footnote 1 Many of our procedures have benefits for other research areas and methods, but we recognize that we cannot generalize to all institutions and types of work. Our focus is primarily on survey and laboratory experiments.Footnote 2 Second, we discuss three institutions: “a planning document,” “a research group,” and “centralizing data collection.” We do not claim that each is unique to us, but we are unaware of another experimental research group that institutes each in the way that we do.Footnote 3 Third, we do not claim that our approach necessarily leads to higher-quality research per se. However, considering that it has resulted in nearly all graduate students involved finishing their degrees, obtaining jobs, and publishing work, we are confident that it works to some degree.Footnote 4 More important, students have uniformly embraced the institutions. That said, downsides and alternatives clearly exist, and we mention these (e.g., costs of our approach) in the conclusion. We also briefly discuss how our approach fits into two disciplinary-wide initiatives: (1) preregistration, and (2) the data access and research transparency (DA-RT) initiative. This article is an example of one approach to graduate advising in research groups using experimental methods.
This article discusses our experiences in building institutionalized procedures so as to facilitate access and channel collective energy. The central aim of these efforts is to enhance the quality of graduate advising.
PLANNING DOCUMENT
Our group’s proclivity to use experiments means that there are resource constraints. Our department has an experimental laboratory and participant pool, but only a limited number of experiments can be implemented simultaneously. Implementation of survey experiments with samples that are representative of populations of interest often requires nontrivial financial investments, especially for graduate students. Online platforms such as Amazon Mechanical Turk provide relatively inexpensive convenience samples, but researchers must still obtain funding for implementation. Consequently, strong incentives exist to carefully plan a study before data collection to ensure that it is well designed. Interested scholars must gain access, but they also must make optimal use of such access.
The planning process requires following a set of steps. Although the steps are intuitive and, in some ways, follow what one may learn in a research-design course (we assume students already have basic research-design training), we are unaware of any statement that follows these precise steps. The purpose is to move research along—that is, it gives a “formula” for how to proceed with prospecti and dissertations. We have found this to be a useful recipe for generating papers, publications, and especially PhD dissertations. This latter point is critical because writing a dissertation can be a daunting task, and we have found that breaking it down into the following pieces relieves stress and provides a clear roadmap—at least when it involves quantitatively oriented political-behavior work.
The following steps are listed in broad strokes. The appendix provides more detail on each step and a running example from Howat’s research, as well as example materials from Mullinix’s work.
• Big Picture Idea: This is a short (i.e., a few pages) document on the general topic and why it is relevant to understanding social, political, and/or economic phenomena. This document iterates approximately five times. For graduate students working on their dissertation, it is the foundation of what they will study well into the future.
• Detailed Literature Review: This is an exhaustive search of research on the topic with detailed descriptions of specific studies. Ideally, it leads to the identification of multiple potential research directions, some of which are tabled for the future. This is when receiving feedback from the research group (discussed in the next section) is critical insofar as others may be aware of related literature of which the researcher is not. It is particularly useful to interact with those in different fields or disciplines to discover literature. Care also must be taken to ensure that the literature reviewed is from an intellectually and demographically diverse set of authors. It is at this stage that the researcher should identify specific gaps in existing knowledge.
• Research Question(s) and Outcomes: Given the identification of a gap in existing work, the next step is to present a specific question (or questions) to be addressed. This includes identifying the precise outcome variable(s) of interest.
• Theory and Hypotheses: This is the development of a theory and hypotheses to be tested, which often involves accessing distinct literatures—sometimes from other disciplines. Researchers should take their time to derive concrete and specific predictions. As part of this step, potential mediators and/or moderators should be specified. Also, in presenting predictions, researchers must be careful to isolate the comparisons to be used.
• Research Design: The scholar then needs to present a design (Leeper Reference Leeper2011), which includes the following:
∘ Discussion of the designs used by others who have addressed similar questions and how the proposed design connects with previous work. In many cases, the ideal strategy is to utilize and extend prior designs.
∘ Discussion of how such a design will provide data relevant to the larger questions.
∘ Identifying where the data will come from, which includes:
▪ Consideration of the sample and any potential biases.
▪ Detailed measures and where the measures were obtained—that is, where have they been used in prior studies? The measures must clearly connect to the hypotheses, including the outcome variables and mediators/moderators.
∘ In many cases, the design may be too practically complex (e.g., the number of experimental conditions relative to a realistic sample size), and decisions must be made about what can be eliminated without interfering with the study goal.
∘ For original data collection, pretests of stimuli, wording of questions, and so forth are critical to ensure that the approach has content and construct validity.
∘ Issues related to internal and external validity should be discussed.
• Data Collection Document: If the project involves original data collection, a step-by-step plan must be presented so as not to later neglect details such as recruitment and implementation.
This framework organizes the research process into manageable pieces and forces researchers to think through each specific decision, which—in our experience—reduces the likelihood of a design error.
• Data-Analysis Plan: There must be a clear data-analysis plan: How exactly will the data be used to test hypotheses? The researcher should directly connect the design and measures to the hypotheses. This often involves creating a table for each measure and how it maps on to specific hypotheses. Which techniques will be used if data collection is imperfect (e.g., a smaller sample size than expected)? This is a commonly missed step. The plan ensures that the correct data will be collected, and it provides a blueprint of what to do when the data are received.
• Institutional Review Board: When appropriate, the researcher should complete the Institutional Review Board (IRB) process. If the project involves nontrivial deception or other aspects that could prevent IRB approval, we strongly urge the researcher to directly contact an IRB representative early in project development. In many cases, the researcher will work with the IRB to determine what is feasible. In general, researchers should be aware that the process can take weeks or months, and they should start this process as soon as possible (e.g., one project in our lab took approximately nine months to obtain approval). Researchers also should address legal and political considerations where applicable (King and Sands Reference King and Sands2015). An additional benefit of following these steps is that the researcher will have already prepared almost all of the information and documents required for an IRB application.
• Merging the Pieces: At this point, the researcher merges the previous pieces into a single document, which is reviewed in its entirety by the research group and faculty (e.g., the prospectus committee). This is essential because in the process of moving from the larger abstract research question to the specific analysis plan, a disconnect may have emerged. In some instances, the particular design no longer addresses the initial motivating question or clearly fills a gap in the literature.
• Implementation: From here, data are collected and analyzed. The planning document serves as a guide to writing up the results and, potentially, identifying reasons why the researcher may not have found what was expected. An essential part of this is careful recordkeeping of data-analytic choices.
This framework organizes the research process into manageable pieces and forces researchers to think through each specific decision, which—in our experience—reduces the likelihood of a design error. To see how failure to follow these steps can be detrimental, we provide a few examples from our own work in the appendix.Footnote 5 Having researchers follow these steps also helps to manage lab access and financial resources (e.g., for survey experiments) because users are expected to have gone through a process such as this before data collection. A vital complementary component of the process is that it provides clear points at which a researcher—particularly a graduate student taking these steps—can receive feedback. This leads to our next institution, which aims to leverage the collectivity for individual advantage.
A RESEARCH GROUP
We are or have been part of a seven- to 15-person research group that meets bi-weekly. The group is graduate-student–centered and includes the main faculty adviser as well as other faculty members from political science and psychology. Whereas most in the group focus on experimentally based political-behavior work, others who work in related areas (and disciplines) attend. This provides the advantage of exposure to and feedback from alternative perspectives. At each meeting, usually three graduate students present their research at each of the various stages described previously.
There are several benefits of a research group. First, it ensures that PhD students do not become isolated as they embark on the difficult task of developing a dissertation idea (although students regularly also present non-dissertation projects). Second, students benefit from multiple sources of feedback other than their adviser at each stage of a project; moreover, the adviser learns of other perspectives and can advise accordingly. As such, the group can act as an early peer review (Leeper Reference Leeper2015, 13) that includes advice from senior graduate students (Fugate, Jaramillo, and Preuhs Reference Fugate, Jaramillo and Preuhs2001). Third, students discover shared interests and engage in collaborative projects. They frequently share data-collection costs by merging instruments that have common elements (e.g., survey questions), even if the main foci are quite distinct. An example from our experience occurred when one study exploring attitudes toward tax policies joined the instrument of another study investigating opinions about scientific technologies (Bolsen and Druckman Reference Bolsen and Druckman2015; Mullinix Reference Mullinix2016). This approach generates considerable cost savings by using the same respondents and sharing demographic data from the survey.Footnote 6 Fourth, the group serves as a focal point for visitors from other departments, other universities, and/or industry; for us, this has resulted in cross-department and cross-institutional collaborations. For example, graduate students from two universities coordinated experiments embedded in exit polls in multiple locations to increase the diversity and size of their sample (Klar and Piston Reference Klar and Piston2015; Mullinix et al. Reference Mullinix, Leeper, Druckman and Freese2015). Other students collaborated on projects with faculty from other departments (within and beyond the university) and a survey-research firm interested in various methodological questions that cohered with their interests.Footnote 7
What makes our research group relatively unique—compared to typical brown bags and writing groups—is that it is tied to the iterative planning process, is an institutional check on scholars to ensure that they maximize their resources (e.g., lab time and/or survey costs), and has generated collaborations across disciplines and organizations.
CENTRALIZING DATA COLLECTION
A research group in which researchers use similar methods brings with it economies of scale. In our case, group members often use the lab to collect data and/or run surveys on a given population, both of which involve programming questionnaires and overseeing data collection. We hire an undergraduate student to program the surveys, assist with recruitment, and conduct data collection (e.g., in the laboratory or launching personalized e-mail invitations for participation). The student also cleans (i.e., recodes and merges) datasets. This is a collective good for graduate students who otherwise would have scant access to such assistance. In our case, the department recognized the benefit and provided partial financial support. The specifics of the arrangement are as follows:
• An individual—in our case, an undergraduate—works as a “data manager.”Footnote 8 This person is trained in survey programs such as Qualtrics and SurveyMonkey. Via meetings with faculty and graduate-student lab members, the data manager learns how to (1) merge data files (e.g., for multiwave studies, distinct data for different conditions if needed); (2) organize data (e.g., make all responses numeric); and (3) finalize variable names.
• Once the planning document is complete up to the implementation stage, a final copy of the instrument becomes available. The instrument is sent to the data manager, who programs it using the appropriate software. Multiple people proof the instrument.
• The account in which the instrument is programmed is inaccessible to the researcher and other members of the research group. Programs such as Qualtrics allow researchers to program their own survey (if needed), share the instrument with the data manager, and then implement the study on the data manager’s account so that researchers do not have access to the data or the instrument while the data are being collected.
• Data collection occurs; researchers can be kept informed of the number of respondents, but they have no access to the data (i.e., they cannot assess the data or “change” the instrument).Footnote 9
• The researcher sends a data key to the data manager. The data manager organizes, cleans, and finalizes the data into a file.
• The uncontaminated data are delivered to the researcher and, if a graduate student, the adviser as well.
A research group in which researchers use similar methods brings with it economies of scale.
This process benefits the data managers insofar as they learn about the research process. Indeed, in multiple cases, the undergraduate data managers ultimately collaborate on projects, which results in publications (Busby, Druckman, and Fredendall Reference Busby, Druckman and Fredendall2017; Druckman, Fein, and Leeper Reference Druckman, Fein and Leeper2012; Druckman et al. Reference Druckman, Lynn Hennessy, Charles and Weber2010). This process also facilitates the merging of projects. Group members have no access to the data before data collection is complete, which is generally good research practice (e.g., it makes clear that the researcher could not have altered the original data).Footnote 10 Our data centralization is similar to what Lin and Green (Reference Lin and Green2016) called a research-group’s standard operating procedure insofar as it codifies our group’s “standing decisions” on certain issues. Yet, it does not go as far as their suggestion of a standing document on data-analysis decisions—which we hope to explore and potentially adopt in the future.
Certainly, not all researchers have the resources to hire a formal data manager. In these situations, we encourage them to institutionalize a practice whereby a third party (e.g., a graduate student or a faculty member) simply has access to the raw “uncontaminated” data as they are received. For example, a researcher can program a survey in Qualtrics, share the link with a colleague or a student, and then send the data directly into that third party’s Qualtrics account. The third party does not have to do much other than simply let the data roll into their account. In an era in which there are increasing concerns about scientific fraud (Simonsohn Reference Simonsohn2013), this approach protects the researcher and the integrity of the research; as such, it should be considered “best practices.”
CONCLUSION
We outline three institutions established to enhance graduate advising in the context of survey- and laboratory-based experimental research (focused on political-behavior research) at a PhD-granting university. The purpose was to most productively allocate lab space and funds for surveys; create collective benefits and synergies; and, ultimately, enhance the quality of graduate-student research. Although each institution echoes common practices in the discipline, we are unaware of another statement that ties them together. We cannot assess the counterfactual of what students’ research would look like sans our approach; however, completed and published dissertations and feedback from participants suggest that there is value added.
We suggest that our planning document could be viewed as related to a form of internal preregistration (i.e., disclosure of one’s research design and predictions before a study’s implementation, typically in a public registry). If nothing else, the document should make formal preregistering a relatively easy exercise. For those who advocate for the disciplinary benefits of preregistration (Nosek et al. Reference Nosek2015, 1423), this is a positive externality of the planning document. Additionally, our planning document and data centralization together facilitate compliance with the principles of the DA-RT initiative because it would result in a document with a clear set of procedures, a clean and easily posted dataset, and an analysis plan (Key Reference Key2016; Lupia and Elman Reference Lupia and Elman2014).
It is important to recognize the limitations and downsides to our approach. To be clear, the usefulness of our specific approach may not be consistently generalized beyond survey and lab-experimental research groups. For instance, our planning document does not include a place per se for fieldwork or more inductive qualitative approaches. Our efforts to blind researchers to data-collection specifics could be problematic for those running field experiments in which the researchers may need to be aware of compliance and consider approaches to equalize it across conditions. Additionally, the existence of a lab and concomitant activities draws students with certain proclivities, which could generate “group think” (i.e., the counter to the synergistic energy produced). Moreover, students whose work is not best pursued in an experimental paradigm may find the presence of the group alluring and mistakenly alter their research aspirations. We attempt to temper these possibilities by inviting students who conduct different types of work to our research-group meetings. We also aim to include multiple faculty members and those from outside of political science in an effort to prevent students from creating a narrow research agenda tightly tied to one faculty member (and more generally develop their own scholarly identity). This is an important point for creating the space for genuine innovation and not predetermined incrementalism (Sniderman Reference Sniderman1995). All of this, however, must be counterbalanced with the need to maintain a reasonably sized group to ensure access and feedback.
We consider this brief article to be the start of a conversation. Laboratory facilities are still relatively new for most departments, and financial resources for surveys remain limited. How best to leverage them and enhance the graduate-student experience is far from obvious. Ours is but one of many possible approaches, and we look forward to further discussions on how to best advise graduate students engaged in experimental research.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1049096518000033.
ACKNOWLEDGMENTS
The authors thank Phillip Ardoin, Paul Gronke, and Cindy Kam for their feedback and assistance with this project. We also thank the researchers who participated in our survey and provided information about their own surveys and experimental labs. Their feedback was instrumental not only for this article but also for improving the structure and functioning of our own labs.
