Much of our discipline’s understanding of political attitudes and behavior has been developed through studying two common groups: nationally representative samples and college students. Nationally representative samples are expensive and often lack internal validity; however, by design, they have high external validity. Student samples, although less representative, are often less expensive and can better facilitate experimental designs, providing strong internal validity. In this article, we present colleague crowdsourcing as a complementary research design that leverages strengths of each approach, and we illustrate its worth in a study of presidential-debate effects. We find that crowdsourcing not only facilitated our data collection but also engaged many students in active learning about the debates in ways that they otherwise might not have experienced. Thus, colleague crowdsourcing has benefits for both research and teaching.
COLLECTING DIVERSE LARGE-N DATA IN NATURAL SETTINGS
Collecting large samples of diverse respondents in a natural setting is a challenge for our discipline. Although nationally representative surveys can achieve this end, they are generally very expensive. Students, however, often are willing to participate and are far more affordable. Yet, they present at least two concerns for external validity (Mintz, Redd, and Vedlitz 2006; Peterson Reference Peterson2001).
First, student samples are not representative of general adult populations (Oakes Reference Oakes1972; Sears Reference Sears1986). This concern often is overstated, however, because students tend to resemble adult populations across a range of important covariates, such as partisanship and media use (Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia.2011, 51). Moreover, if scholars are interested in estimating relationships between variables, they can use student samples to create valid inferences—even in cases in which the sample differs substantially from the population. If a treatment effect of interest is homogeneous in the population, any sample can produce an unbiased estimate. However, even if the treatment effect varies, it can be modeled as long as the sample provides variation across the relevant moderating variables. Thus, unbiased estimates of treatment effects require diverse but not representative samples. For example, in the case of presidential debates, the effect of candidate attention to immigration on viewers’ attitudes toward the candidate might depend on a viewer’s ideology and race. In this case, unbiased estimates would depend on obtaining a sufficient number of respondents across the ranges of ideology and race but would not require the sample’s percentage of conservatives or African Americans (for instance) to equal those in the population (Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia.2011). Many single-campus student samples may lack this needed variation.
Second, student-based studies generally are conducted in artificial settings—often a computer lab. Laboratory environments tend to eliminate distractions, resulting in treatment effects that are larger than those in natural settings (Jerit, Barabas, and Clifford Reference Jerit, Barabas and Clifford2013). One solution is to allow participation in more natural settings (Kinder Reference Kinder2007) in which distractions introduce variation in participant attentiveness (e.g., Albertson and Lawrence Reference Albertson and Lawrence2009). However, technological and logistical limitations often impede this approach.
Crowdsourcing data collection can mitigate both concerns. A relatively new concept in business and an even newer concept in academia, crowdsourcing is “a strategic model to attract an interested, motivated crowd of individuals capable of providing solutions superior in quality and quantity to those that even traditional forms [can]” (Brabham Reference Brabham2008). Footnote 1 Our approach, described in detail below, builds on crowdsourcing work by reaching out to the political science community to access a more diverse student-respondent pool participating in more natural settings. Of added benefit, this approach provides instructors with resources to facilitate classroom discussions—and may even heighten student engagement in the political process.
COLLEAGUE CROWDSOURCING FOR THE 2012 PRESIDENTIAL DEBATES
Our substantive interest is to understand how candidate debate behaviors affect viewers’ attitudes (Boydstun et al. Reference Boydstun, Glazier, Pietryka and Resnik2014). Despite the salience and visibility of presidential debates (Benoit, Hansen, and Verser Reference Benoit, Hansen and Verser2003; Jamieson and Birdsell 1990; Marcus and Mackuen Reference Marcus and Mackuen1993), few studies have collected real-time reactions that allow for the study of individual debate moments; those that have done so use very small samples (e.g., Fridkin et al. Reference Fridkin, Kenney, Allen Gershon, Shafer and Woodall2007; McKinney and Rill Reference McKinney and Rill2009; Pfau, Houston, and Semmler 2005).
Participation far exceeded our expectations, with respondents from all 50 states, the District of Columbia, Puerto Rico, and even outside of the United States.
Thus, we set out to measure debate reactions using a web application, or “app,” that we designed for use on smartphones. Footnote 2 The app was also accessible from tablets and personal computers, allowing viewers to react to the debates in real time from anywhere with Internet connectivity. A screenshot of this app, React Labs: Educate, is displayed in figure 1. Respondents used the app while watching the debates live, indicating (at any time they wished) whether they “agreed” or “disagreed” with the candidates and whether they thought the candidates were “spinning” or “dodging” the question.
Figure 1 React Labs: Educate App Interface
(Color online)
We needed a larger, more diverse sample of app users than any of our campuses could provide in isolation or combined. Therefore we targeted our recruitment efforts at instructors across the country, knowing that they are uniquely able to encourage student participation (e.g., in exchange for extra credit). To encourage instructors to register their classes and promote participation, we designed an incentive package aimed at helping them to achieve some of their own teaching and learning goals.
The materials we provided to registered instructors are available on the project website (http://reactlabseducate.wordpress.com). Before the debates, registered instructors received the following materials:
-
• PowerPoint slides and lecture notes covering the history of presidential debates—including YouTube links to memorable debate moments as well as research on debate rhetoric, debate strategies, and debate effects
-
• discussion questions
-
• a list of resources, websites, and research collections on presidential campaigns and debates
-
• citations and abstracts of relevant debate research
-
• alternative assignments for students unable to watch the debates live
After the debates, registered instructors also received the following:
-
• Within 12 hours of each debate: presentation-ready PowerPoint slides with preliminary results from respondents who used the app
-
• After the final debate: for each debate, a list of their students who participated
These resources linked political science teaching and research, helping instructors discuss the debates in a way that connected theory with contemporary politics.
We recruited instructors by sending more than 120 individual e-mails inviting colleagues to participate in the project and by sending invitations to key listservs and blogs. Footnote 3 Instructors registered their classes to participate through the project website. Each registered course was assigned a unique course identification number, which enabled us to send instructors confirmation of their students’ participation but also required us to send a unique e-mail with instructions and the course identification number for each registered class. This challenge was made easier by Gmail’s Mail Merge, which allowed us to merge e-mail addresses, course identification numbers, instructor names, and course names from a database into individual e-mails, thereby automating the process of sending individualized messages. Footnote 4
We embedded a predebate survey in the app itself and used a paid (but relatively inexpensive) subscription to SurveyMonkey® to administer a postdebate survey. Survey Monkey® provided the capacity to handle a high volume of student participants, to ask a large number of follow-up questions, and to download the results in a spreadsheet.
Following through on our promise to provide next-day figures and preliminary results proved challenging. We offered our graduate students free food and good cheer to stay up all night after each debate, crunching numbers and compiling PowerPoint slides. Although the process was labor intensive, we felt that providing instructors with immediate results that they could use in class to facilitate discussions of the debates was a critical incentive for participation.
Our research design represents a major advance in external validity. In terms of representativeness, the app allows us to draw on a large and diverse enough sample to include the variation we need for analysis. In terms of artificiality, the app allows students to participate in the study from wherever they would normally watch a debate (e.g., home, a friend’s house, or a debate-watch party).
RESULTS
Participation far exceeded our expectations, with respondents from all 50 states, the District of Columbia, Puerto Rico, and even outside of the United States. In total, 263 instructors registered at least one course to participate in at least one debate, representing 361 courses and more than 13,000 potential student respondents. Footnote 5 Across the three presidential debates and one vice presidential debate, almost 5,000 undergraduates participated at least once. Footnote 6 Counting each respondent in each debate separately, the app received 8,006 respondents, the demographics of which are summarized in table 1.
Table 1 Study Demographics Compared to National Demographics
Notes: App estimates include all 8,006 participants across the four debates, including those who participated in more than one debate. The numbers do not total 8,006 on any given demographic item due to non-response on that item.
a National estimates are from the US Census.
b National estimates are from the Pew Research Center for the People & the Press, October 2012, accessed January 23, 2013, from the iPOLL Databank, The Roper Center for Public Opinion Research, University of Connecticut. Available at http://www.ropercenter.uconn.edu/data_access/ipoll/ipoll.html. Footnote 7
c National estimates are from the 2008 American Religious Identification Survey.
d National estimates are from the 2012 American Community Survey One-Year Estimates.
As table 1 illustrates, our sample is similar to national population means for gender, income, race, party identification, and religion. The major demographic difference is in age because our recruitment efforts were targeted at college undergraduates. Although the sample is not nationally representative, nonetheless we received more than 175 participants in each age group, allowing us to estimate debate effects that vary with age. In terms of both representativeness and variation across a range of variables, these data represent major progress in sample quality over single-campus convenience samples. Table 2 illustrates this variation in more detail.
Table 2 Participant Frequencies by Ideology and Race/Ethnicity
Notes: Ideology and race were measured in the predebate survey. Ideology was measured with a 100-point sliding scale ranging from 0 (extremely liberal) to 100 (extremely conservative). In the table, participants scoring between 0 and 39 on this scale are classified as liberal, between 40 and 60 as moderate, and between 61 and 100 as conservative.
Part A of table 2 displays the number of students who took part in the debate study, categorized by ideology and race/ethnicity. The table shows that the large number of respondents provided a sufficient number in each cell to model heterogeneous treatment effects—even for those cells that captured rare combinations (e.g., conservative African Americans).
For comparison, part B of table 2 shows the same breakdowns for ideology and race/ ethnicity compiled from the five courses in which students participated from a single campus (University of California, Davis). There are only three African Americans in the UC Davis sample, none of whom identify as conservative, thereby preventing the estimation of heterogeneous treatment effects for this group. This data binning problem occurs across a range of demographic and attitudinal measures.
We view the teaching benefits of our study—providing instructors with easy-to-use classroom materials and a method by which to actively engage students in the political process—as a hopeful indication that the colleague-crowdsourcing approach can facilitate a symbiotic relationship between teaching and research.
Thus, our crowdsourcing approach realized several benefits over traditional, single-campus, fixed-location research studies. Although the sample is not representative and app users may have been paying closer attention to the debates than typical viewers, this approach allowed us to collect data in more natural settings than previously possible. It also enables estimates of treatment effects across a range of covariate profiles that otherwise would be inaccessible. Therefore, the sample cannot provide an unbiased estimate of the prevalence of a certain trait in the general population, but it is uniquely suited to produce estimates of many different treatment effects.
THE TEACHING AND LEARNING BENEFITS OF CROWDSOURCING
In addition to the methodological and logistical benefits of our crowdsourcing approach, our solution facilitated teaching and learning. Because of their salience and scale, presidential debates represent key opportunities to encourage student engagement with the political process, which can improve political knowledge and civic skills—especially among those with lower initial levels of political interest (Beaumont et al. Reference Beaumont, Colby, Ehrlich and Torney-Purta2006). When instructors highlight engagement and civic themes, their students’ future political engagement and voter turnout increase (Hillygus Reference Hillygus2005; McCartney, Bennion, and Simpson Reference McCartney, Bennion and Simpson2013). Furthermore, watching debates tends to boost political efficacy, trust, and information among youth while decreasing cynicism (Kaid, McKinney, and Tedesco 2007; McKinney and Rill Reference McKinney and Rill2009). Many of our student participants likely would not have watched the debates were it not for the app and the incentives that we encouraged instructors to offer. Even for those students who would have watched anyway, using our app turned watching TV—a generally passive activity—into an interactive experience. Extensive research has demonstrated that active learning techniques improve test scores (McCarthy and Anderson Reference McCarthy and Anderson2000), engagement with the material (Brown and King Reference Brown and King2000; Hess Reference Hess1999; Ruben Reference Ruben1999; Wolfe and Crooktall Reference Wolfe and Crooktall1998), learning (Pace et al. Reference Pace, Bishel, Beck, Holquist and Makowski1990; Perry Reference Perry1968; Sutro Reference Sutro1985; Washbush and Gosen Reference Washbush and Gosen2001), and interest (Hess Reference Hess1999; Smith and Boyer Reference Smith and Boyer1996). Although we do not directly measure these effects here, the literature leads us to expect that using the app aided student learning.
Our crowdsourcing method benefited instructors as well. During the month of October 2012, our publicly available webpage featuring overnight result summaries was accessed more than 5,000 times. In addition to the result summaries, participating instructors accessed our password-protected teaching-resources webpage 450 times. We view the teaching benefits of our study—providing instructors with easy-to-use classroom materials and a method by which to actively engage students in the political process—as a hopeful indication that the colleague-crowdsourcing approach can facilitate a symbiotic relationship between teaching and research.
THE FUTURE OF COLLEAGUE CROWDSOURCING
We believe colleague crowdsourcing holds considerable promise for future studies, particularly in light of ongoing technological innovations, which make national (or even international) crowdsourcing increasingly feasible. Our app facilitated crowdsourcing by enabling participation across the country, but there are many other potential uses of colleague crowdsourcing; we certainly do not expect all scholars to create an app.
For example, colleague crowdsourcing might be used to foster large-scale and geographically diverse participation in studies using survey platforms such as Qualtrics® and SurveyMonkey®—or participation by specific target groups, such as first-generation college students or Muslims. Colleague crowdsourcing could be used to collect simple cross-sectional survey data, panel data during the course of an academic term, or data derived from survey experiments. It also could be used to measure aspects of the political environment (e.g., counting yard signs or political bumper stickers). In addition, we can imagine the incentive portion of the crowdsourcing approach taking many forms, including access to the data, webcast guest lecturers, and research notes on the findings for use in class. With enough lead time to include information about a study in their syllabi and/or to incorporate time for discussion in their lecture plans, many instructors may be keen to encourage student participation in an interesting study. In short, the crowdsourcing approach as a recruitment technique is flexible and scalable. Overall, new research technologies coupled with colleague crowdsourcing create a rich opportunity to incorporate research methods, local and global findings, and temporally relevant data in the classroom in a way that can aid research efforts while stimulating a new level of active learning.
ACKNOWLEDGMENTS
We are immensely grateful to the hundreds of instructors and thousands of students who participated in this crowdsourcing exercise; our colleagues Debra Leiter, Jack Reilly, and Michelle Schwarze, who helped develop the teaching resources we used to encourage participation and who sacrificed themselves for our postdebate all-night data-crunching sessions; and our colleague Philip Resnik at the University of Maryland, who conceived of the mobile reactions platform and with whom we collaborated to make React Labs: Educate a reality. We presented a previous version of this article at the 2013 APSA Teaching and Learning Conference. v
