Meta-analyses suggest that political advertising does not affect participation (Lau and Rovner Reference Lau and Rovner2009; Lau, Sigelman and Rovner Reference Lau, Sigelman and Rovner2007). Other work finds that the effects depend on message timing (Doherty and Adler Reference Doherty and Adler2014; Krupnikov Reference Krupnikov2011) and characteristics of the messenger (Krupnikov and Bauer Reference Krupnikov and Bauer2014; Krupnikov and Piston Reference Krupnikov and Piston2015). A possibility that has not been directly explored is that participants in survey and lab experiments do not respond to campaign messages in a way that reflects responses to messaging encountered in day-to-day life (Arceneaux Reference Arceneaux2010; Barabas and Jerit Reference Barabas and Jerit2010; Gaines, Kuklinski and Quirk Reference Gaines, Kuklinski and Quirk2007; Kinder Reference Kinder2007; Shadish, Cook and Campbell Reference Shadish, Cook and Campbell2002).

We present findings from a survey experiment that mirrors a pair of field experiments reported by Doherty and Adler (Reference Doherty and Adler2014). Those field experiments – conducted in Colorado state Senate districts in August and October of 2012 – found that both positive and negative partisan mailers increased reported intent to turnout among the unaffiliated likely voters they targeted and surveyed by phone shortly after the mailers were sent.Footnote ¹ Results of our survey-based experiment, which used images of the mailers as stimuli, indicate that exposure to these messages depresses intent to turnout among likely Independent voters. The findings demonstrate that field and survey-based experiments that look remarkably similar on their face can yield divergent answers to a seemingly simple question: Does political advertising mobilize or demobilize Independent voters?

Survey participants were US residents recruited through Amazon’s Mechanical Turk interface and were paid $0.50 to complete the survey about two weeks after the second field experiment (November 1–4, 2012; sample characteristics reported in Table A2 of Supplementary material). After answering a series of demographic and political questions, 80% of respondents were randomly assigned to view one of the four mailers from the first field experiment reported by Doherty and Adler (Reference Doherty and Adler2014), with the remainder serving as a control group. Two mailers were positive in tone and emphasized favorable characteristics of the sponsoring (Republican) candidate; two attacked the (Democratic) opponent. All respondents were then asked how likely they were to vote in the 2012 general election on a scale ranging from 1 (will definitely not vote) to 4 (will definitely vote).Footnote ²

A total of 874 individuals who did not report already having voted began the survey.Footnote ³ We wish to mirror the characteristics of the field experiment, so we excluded 186 respondents who, prior to treatment, indicated they were not registered to vote, as well as 55 respondents who did not provide responses to the items used in our analysis.Footnote ⁴ This yields a sample of 633 respondents. We stack these data with data from field experiments and estimate models predicting intent to turnout, interacting the treatments with indicators for each experiment. We also estimate analogous models to improve the comparability of our respondents and the Independent likely voters targeted in the field experiments by excluding: (1) 169 respondents who reported not having voted in 2008 and/or 2010; (2) 425 who identified as Democrats or Republicans; (3) both of these groups.Footnote ⁵ We report estimates from these models in Figure 1.

Figure 1 Effects of mailer treatments on intent to turnout: Markers are from estimates reported in Table A7 of Supplementary material. Whiskers are 95% confidence intervals. Markers with gray confidence intervals are from models restricting the survey experiment sample to likely voters (those who reported voting in 2008 or 2010). The estimated effect of the negative mailer treatment in each field experiment is statistically distinguishable from the effect in all survey experiment sub-samples. The effect of the positive mailer treatment in field experiment 1 is not distinguishable from the effect in any survey experiment sub-sample (p < 0.05 for all comparisons); the effect in field experiment 2 is statistically distinguishable from all survey experiment sub-samples (p < 0.05), except Independent likely voters (p = 0.078). See Table A7 in Supplementary material for details.

The solid markers show that both types of mailers increased intent to turnout in the field experiments (p = 0.211 for positive mailer in first experiment). In contrast, the point estimates from the survey experiment all indicate that the mailers depressed turnout intentions. The estimated effect of the positive mailer from the second (but not first) experiment is statistically distinguishable (p < 0.05) from the effect in the survey experiment (with the exception of the estimate restricting the sample to Independent likely voters). The estimated effect of the negative mailer in the survey experiment is distinguishable from the effect in each field experiment across all sub-samples. Strikingly, in the survey experiment we found that the negative mailer significantly depressed intent to turnout among Independents – those targeted in the field experiments. This is true among all Independents (p < 0.01; β = −0.413), as well as among likely Independent voters (p < 0.05; β = −0.319). In each case, these estimated effects are substantial (approximately 1/2 standard deviation).

Because we did not randomly assign individuals to a survey or field experiment condition (e.g., Jerit, Barabas, and Clifford Reference Jerit, Barabas and Clifford2013), we could not directly determine why these differences emerge. In Table 1 we compare key characteristics of the designs. Existing evidence suggests that the demobilizing effects of negative advertising late in a campaign should not emerge in our survey experiment given that participants were unlikely to have heard of the candidates identified in the mailers (Krupnikov Reference Krupnikov2011). Similarly, the fact that the gap between treatment and measurement of the outcome was larger in the field experiments suggests that the effects should be attenuated – rather than boomerang – in those designs (e.g., Gerber et al. Reference Gerber, Gimpel, Green and Shaw2011). Although the survey mode used to measure our outcome differed across the experiments (IVR vs. online), respondents did not speak to an in-person interviewer in either case.

Table 1 Comparison of experimental designs

^† The second field experiment also targeted voters in SD 35 and included an additional treatment condition (a contrast mailer that included positive information about the sponsoring candidate and an attack on their opponent). Although substantively inconsequential, we excluded respondents from SD 35 and the contrast mailer conditions to maximize comparability across designs.

^‡ Respondents who reported that they had already voted or were not registered to vote were excluded from all models.

Educational attainment amplifies the negative effect of the negative mailer in the survey experiment, (columns 3 and 4 of Table A4 in Supplementary material), but the districts targeted in the field experiments are better educated than the national public (see Table A8 in Supplementary material), making it unlikely that sample characteristics explain the divergence (Krupnikov and Levine Reference Krupnikov and Levine2014; Mullinix et al. Reference Mullinix, Leeper, Druckman and Freese2015). A pre-treatment measure of how much recent political mails participants in our experiment had received does not moderate the treatment effects. Field experiment participants who responded to the survey may have been more politically engaged, but those who did were only 5%–6% points more likely to turnout than those who did not. Additionally, Mechanical Turk respondents who opt in to political surveys tend to be more interested in politics than the general public (Berinsky, Huber, and Lenz Reference Berinsky, Huber and Lenz2012), and likely voters in the survey experiment did not respond to the treatments in a distinctive fashion.

Ultimately, we suspect the explanation for the pattern we found is tied to research design. People may respond differently to communications when they know their responses to information are being studied. For example, people may try to adhere to perceived social norms prescribing aversion to political messaging. Pre-treatment political questions commonly used in survey experiments, such as the one we fielded, may also lead participants to think about the messages differently than they would if they were exposed to them as they, say, sorted through their mails. Whatever mechanisms drove our divergent findings, our evidence demonstrates that survey and field experiments conducted at approximately the same time using identical stimuli and outcome measures can yield conflicting conclusions.