Scholars argue that organized groups have substantial effects on policy outcomes (e.g., Gilens and Page Reference Gilens and Page2014). Scholars have posited multiple mechanisms to explain their influence. In this paper, we provide a unique test of one hypothesized mechanism: so-called outside lobbying (e.g., Kollman Reference Kollman1998), in which organized groups attempt to affect public opinion on a policy.Footnote 1 In particular, organized groups frequently deploy public advertising, especially on television, to try to advance their policy priorities. For instance, during the 108th Congress, Falk, Grizard, and McDonald (Reference Falk, Grizard and McDonald2005) estimate that interest groups spent $320 million on television issue advertising (2021 dollars). This sum is smaller than the $1.3 billion that groups spent on television advertising in the 2020 presidential election (Ridout, Fowler, and Franz Reference Ridout, Fowler and Franz2021), but still substantial. However, the effects of such advertising on public opinion are difficult to ascertain.
We report what we believe to be the first field experiment on how television advertising affects public opinion on issues. Our experiment is highly unique: despite the enormous sums outside groups spend on television ads to influence voters’ views on candidates and issues, there is only one prior published field experiment on the effects of television ads on public opinion, which considers its effects on candidate choice (Gerber et al. Reference Gerber, Gimpel, Green and Shaw2011). We examine the effects of four television advertisements on voters’ issue attitudes, issue knowledge, and intent to engage in political activism. The advertisements cover immigration and LGBTQ nondiscrimination, two salient topics subject to considerable outside lobbying over the last decade.
We find that television ads can have effects on public opinion while the ads are airing and that the ads can teach voters facts they remember, contrasting with prior findings on candidate campaign ads (Huber and Arceneaux Reference Huber and Arceneaux2007). However, we find that not all ads persuade and that the ads that do persuade have effects that fade rapidly, consistent with findings from candidate campaigns (Gerber et al. Reference Gerber, Gimpel, Green and Shaw2011; Hill et al. Reference Hill, Lo, Vavreck and Zaller2013; Kalla and Broockman Reference Kalla and Broockman2018). In short, we find that television advertising can allow groups to temporarily change public sentiment and to inform the public but that not every ad is effective and that persuasive effects may be short-lived.
Advertising’s Persuasive Effects and Interest Group Outside Lobbying
Field experiments testing the effects of political television advertisements are rare. Despite the vast sum outside groups spend on television advertising for election campaigns and legislative fights, this experiment represents one of the only field experiments on persuasive political television advertising ever reported, outside of Gerber et al.’s (Reference Gerber, Gimpel, Green and Shaw2011) pioneering study (see also natural experiments from, e.g., Huber and Arceneaux Reference Huber and Arceneaux2007; Spenkuch and Toniatti Reference Spenkuch and Toniatti2018). Moreover, prior literature on the effectiveness of television advertising in American politics has largely focused on its effects on candidate choice (but see Hall and Reynolds Reference Hall and Reynolds2012). In this experiment, we build on this literature by testing the effect of television advertising on voters’ views on issues.
Prior theoretical and empirical work would suggest that television issue advertising may be more persuasive than candidate campaign advertising. First, issue advertising may be less likely to encounter “partisan resistance” than candidate ads (Zaller Reference Zaller1992): whereas candidates in partisan elections have a “D” or “R” next to their name, issue positions do not, and voter knowledge of which party is associated with various issue positions is imperfect. Furthermore, in outside lobbying campaigns, the public often only hears advertising from one side while candidate campaigns are more frequently two-sided (Falk, Grizard, and McDonald Reference Falk, Grizard and McDonald2005), potentially producing larger persuasive effects (Zaller Reference Zaller1992). Consistent with this expectation, in their meta-analysis of persuasion field experiments, Kalla and Broockman (Reference Kalla and Broockman2018) find that campaign outreach in the form of canvassing and direct mail is substantially more persuasive in ballot measures contests (i.e., on people’s views on issues) than in partisan candidate elections. Given this existing theoretical and empirical work, outside lobbying television advertising may be expected to produce larger persuasive effects.Footnote 2
On the other hand, there is ample room for pessimism: Gerber et al. (Reference Gerber, Gimpel, Green and Shaw2011) found that television ads’ persuasive effects on candidate choice rapidly decayed, and Huber and Arceneaux (Reference Huber and Arceneaux2007) found no effects of presidential ads on factual knowledge (although they do find evidence of persuasion).
Experimental Design
We test the effect of four advertisements in two issue domains, immigration and LGBTQ nondiscrimination laws. Both issues have attracted significant public attention over the last several years and featured heavily in the 2020 US Presidential campaign. Figure 1 and Table OA1 summarize the experimental design. We report more information on the experimental setting in the Online Appendix. Replication data are available in Kalla and Broockman (Reference Kalla and Broockman2021).
Figure 1. Experimental Design
Baseline Survey
To measure the effects of these advertisements, we conducted a preregistered randomized experiment and survey measurement using the design in Broockman, Kalla, and Sekhon (Reference Broockman, Kalla and Sekhon2017).Footnote 3 The experiment began by recruiting registered voters (n=1,082,605) via mail for an ostensibly unrelated online baseline survey, presented as the first in a series of surveys about political and social topics. These registered voters all subscribed to particular TV providers with the technological ability to target television advertising at the household level. The experiment occurred in California, Colorado, Michigan, North Carolina, Tennessee, and Wisconsin.
A total of 32,923 voters responded to this baseline survey, during which we gathered pretreatment covariates for improved precision and respondents’ email addresses to invite them to an endline survey. From this survey, we constructed an index of respondents’ baseline opinions related to immigration, LGBTQ issues, and partisan political views. Due to budget constraints, at this stage, we removed the 1,501 individuals who were already in the most supportive deciles of both the immigration and LGBTQ indices.
Experimental Conditions and Stimuli
We next block-randomly assigned baseline survey respondents at the household level to one of three conditions: (1) a group that received two immigration ads (“Eddie” and “Prosperous Future”; n=10,467), (2) a group that received two immigration ads (“MariCruz/Kandy” and the same “Prosperous Future” ad; n=10,468), and (3) a group that received an ad about LGBTQ rights (“Open to All”; n=10,469).
Given space constraints, the Online Appendix describes the ads in detail. Briefly,
-
• The “Prosperous Future” immigration ad features a middle-aged white woman sharing how she used to think all immigrants should just “get in line,” but then she learned how the immigration system is broken.
-
• The “Eddie” ad features a first-person narrative from an asylum seeker. He shares how he holds many American values (family, hard work, and freedom) and is a business owner. He discusses how he supports immigration and asylum reform.
-
• The “MariCruz/Kandy,” ad features two coworkers, one white (Kandy) and the other Latina (MariCruz). MariCruz shares how she came to the United States as an undocumented immigrant. The ad ends with Kandy sharing that she was surprised to learn that undocumented workers pay taxes. This fact is also displayed on screen.
-
• The “Open to All” ad features an older Christian couple. They describe themselves as small business owners who believe that treating people how they want to be treated is both good for business and required by their Christian faith. They state that nobody should be refused services for being LGBTQ.
The television advertisements were all created by immigration and LGBTQ organized groups and communication professionals.
Treatment Implementation and Outcome Measurement
The advertisements aired for three weeks, a length of time the partner organizations thought would be sufficient to test the ads’ persuasive power. The advertising firm did not stipulate particular networks or hours for the ads to run. Instead, they could run whenever the television was turned on. Across all voters, the average household was exposed to the ads 19.7 times. Put in terms of Gross Rating Points (GRPs), which are defined as 100 times the expected number of times an individual in the target audience viewed the ad, the intervention was therefore equivalent to approximately 1,970 GRPs over the course of three weeks—a large volume. (By contrast, Gerber et al. [Reference Gerber, Gimpel, Green and Shaw2011] randomized media markets to receive up to only 1,000 GRPs per week.) The firm was also able to collect data on how often each household was exposed to an advertisement for a nonrandom 51% of voters who have newer television technologies, allowing us to estimate treatment-on-treated (TOT) effects among this subgroup. We do not know who in a household may have seen the advertisement.
After the advertisements aired, we conducted an ostensibly unrelated posttreatment survey to measure their persuasive effects. This survey made no mention of the specific ads nor was it limited to immigration or LGBTQ issues, but instead it included many unrelated questions to reduce the potential for demand.
To measure how quickly any effects decayed, we randomly assigned the timing of this posttreatment survey. We randomly assigned one third of the respondents to receive an invitation via email to take the survey while the advertisements were still airing (starting two days before the advertising ended), one third to receive an invitation one day after the advertisements stopped airing, and one third to receive an invitation three days after the advertisements stopped airing. Respondents generally complied with their assignment to the survey timing: 57% completed the survey on the day they were invited and 92% within two days.
The advertisements sought to change public opinion toward LGBTQ people and immigrants along two dimensions within each issue domain: increasing support for more inclusionary government policies and decreasing prejudice.
To measure these constructs, the posttreatment survey included multiple items measuring immigration prejudice (eight items), immigration policy (three items), LGBTQ prejudice (four items), and LGBTQ policy (six items). As we preregistered, we combine these items into four indices as well as two overall LGBTQ and immigration indices containing all 11 immigration and 10 LGBTQ items. Following our preanalysis plan, we formed these indices by taking the first dimension from a factor analysis of the appropriate items and recoded all indices such that positive values indicate the intended effect of greater public support. We rescale all indices to have mean 0 and standard deviation 1.
In addition to these indices, we also asked respondents how likely they would be to take various political actions (e.g., contact your Congressperson) about each issue on a 1 (not at all likely) to 5 (extremely likely) scale.
We also asked two items as manipulation checks to assess whether the advertisements were successfully delivered and memorable. We asked whether respondents recalled seeing advertisements on television about several topics, with separate items for immigration and LGBTQ advertisements. We refer to these as measures of advertisement recall (see Figure OA1 for wording).
We also asked about factual knowledge of the immigration system. The “MariCruz/Kandy” immigration advertisement mentioned that undocumented workers pay taxes several times. To measure whether respondents learned this, we presented a list of potentially true facts and asked respondents which they believed to be completely true. One statement was about undocumented immigrants paying taxes. We refer to this as a measure of knowledge (see Figure OA2 for wording). (We did not ask similar questions for the other advertisements because they did not provide particular facts in the same way.)
To estimate the effect of the two immigration ad conditions, we use the LGBTQ ad condition as the comparison group. Similarly, to estimate the effect of the LGBTQ ad, we pool together the two immigration ad conditions and use these respondents as the comparison group. Implicit in this approach is the assumption that an LGBTQ ad has no effect on immigration attitudes and vice-versa. Coppock and Green’s (Reference Coppock and Green2021) results support this assumption, finding no spillover of causal effects on opinion between issue areas.
To estimate treatment effects, we regress the outcomes on the treatment indicators and preregistered pretreatment covariates, which we include to increase precision. Our analysis clusters standard errors at the household level. The treatment effects we estimate are intent-to-treat (ITT) effects among all individuals randomly assigned to receive the advertisements. We also report TOT estimates by limiting our analysis to only those individuals in households with technology recording their viewership and who this technology indicates were shown the advertisements at least once.
The Online Appendix includes a discussion of ethical considerations; full question wordings; representativeness assessments (Table OA2), including baseline values showing this sample is not already at a ceiling; tests of design assumptions (Tables OA3-5); and treatment effect heterogeneity analyses, including by political knowledge and partisanship (Tables OA15–28; 38–39).
Results
Effects of LGBTQ Ad
We report the main results for the effects of the LGBTQ ad in Figure 2. The top panel shows the ITT effects, which compare the entire treatment and control groups. The bottom panel shows the TOT effects among the nonrandom subset of households for which information on actual ad exposure is available.
Figure 2. Estimated Treatment Effects of LGBTQ Ad
Note: Standard errors (thick lines) and 95% confidence intervals (thin) surround point estimates. See Tables OA29–32, 41 for numerical estimates.
We first find large effects on recall of seeing an ad about LGBTQ people. This confirms that the ads were delivered to the treatment group and demonstrates that the ads were memorable. In particular, among all posttreatment survey respondents (regardless of when they were surveyed), we estimate a statistically significant 5.9 percentage point ITT effect on recall (SE = 0.4, p < 0.001). This effect does not appear to decay; three days after the advertisement stopped airing, we still find a 6.0 percentage point increase in recall (ITT: SE = 0.7, p < 0.001). Figure 2 shows that both the ITT and TOT effects are meaningfully sized when expressed in terms of standard deviations.
We also find that the advertisements decrease prejudice against LGBTQ people and increase support for LGBTQ-inclusive policies while the advertisement is airing. However, these effects appear to rapidly decay once the advertisement stopped and are primarily driven by Democratic respondents (Table OA38). First, on the overall index that includes both the prejudice and the policy items, we estimate a statistically significant 0.036 standard-deviation ITT treatment effect while the advertisement is airing (SE = 0.011, p = 0.001; see Table OA37 for estimates by individual items), a result that is robust to multiple testing corrections (Table OA40).
However, these effects did not appear to persist after the ads stopped. One day after the ad stopped airing, we estimate a small, negative, and statistically insignificant effect (ITT: d = –0.007, SE = 0.012, p = 0.52). We estimate a similarly small effect three days after the advertisement stopped (ITT: d = 0.004, SE = 0.012, p = 0.73). The results are similar when examining only the prejudice or only the policy outcome and in the TOT results.
Figures OA8–11 show heterogenous effects by the number of exposures to the ads (i.e., dosage).
Finally, we find no statistically significant effects of the advertisement on respondents’ self-reported likelihood of taking political action, including among respondents in the baseline survey who were most supportive of LGBTQ rights. See Tables OA33–36 for these results.
In summary, the results of the LGBTQ ad suggest that issue ads can have effects on public opinion. However, similar to findings for candidate ads (Gerber et al. Reference Gerber, Gimpel, Green and Shaw2011; Hill et al. Reference Hill, Lo, Vavreck and Zaller2013), we find that these effects decay rapidly.
Effects of Immigration Ads
The main results of the immigration advertisements are reported in Figure 3.
Figure 3. Treatment Effects of Immigration Ads
Note: Standard errors (thick lines) and 95% confidence intervals (thin) surround point estimates. See Tables OA6–10 and OA41 for numerical estimates.
First, like the LGBTQ advertisement, we find meaningful effects on ad recall. Among all posttreatment survey respondents, we find a statistically significant 3.7 percentage-point increase in recall among those who were shown the “Eddie” and “Prosperous Future” advertisements (ITT: SE = 0.6, p < 0.001) and a statistically significant 4.8 percentage-point increase in recall among those who were shown the “MariCruz/Kandy” and “Prosperous Future” advertisements (ITT: SE = 0.6, p < 0.001). These recall effects also do not appear to decay in either condition.
Second, we find that the “MariCruz/Kandy” advertisement taught viewers new information, specifically that undocumented immigrants pay taxes. We find a statistically significant increase in belief that this fact is true of 4.7 percentage points among respondents in the “MariCruz/Kandy” and “Prosperous Future” condition (ITT: SE = 0.8, p < 0.001). This effect also does not appear to decay. Unsurprisingly, given that neither of the other advertisements mentioned this fact, we see no effect on knowledge of this fact in the “Eddie” and “Prosperous Future” condition (-0.4 percentage points, SE = 0.7, p = 0.55).
Despite the immigration advertisements being memorable and imparting new information, there is not clear evidence that these ads had persuasive effects on issue attitudes. On the overall index that includes both prejudice and policy items, we estimate statistically insignificant ITT treatment effects for both the “Eddie” and “Prosperous Future” condition (d = 0.000, SE = 0.007, p = 0.94) and the “MariCruz/Kandy” and “Prosperous Future” condition (d = 0.013, SE = 0.007, p = 0.08).Footnote 4 Examining the TOT effects, the estimated effects on the “MariCruz/Kandy” and “Prosperous Future” condition are substantively small but statistically significant on the overall index (TOT: d = 0.02, SE = 0.0094, p = 0.03). The TOT effects on the separate prejudice and policy indices just fall short of statistical significance (Table OA41). Overall, this suggests there is some possibility that the “MariCruz/Kandy” and “Prosperous Future” condition may have had small effects, but we cannot say with confidence. Effects by respondent partisanship are noisy and inconclusive (Tables OA15–21). On the other hand, the 95% confidence intervals suggest it is unlikely that this condition had an effect any larger than d = 0.04 (top of 95% confidence interval for TOT).
Further consistent with these advertisements not having a persuasive effect, Figures OA3–7 show no evidence of larger effects at higher doses for either advertisement.
Finally, we find no statistically significant effects of the advertisements on respondents’ self-reported likelihood of taking political action, including among respondents in the baseline survey who were most supportive of immigrant rights. See Tables OA11–14 for results.
Discussion
Our experiment finds that outside lobbying television advertising can have effects on public opinion and can impart information people remember. However, not every ad reliably persuades and the persuasive effects that we observe appear to decay rapidly—within a day of the ads being taken off of the air.
Our results suggest several theoretical and substantive implications.
First, our substantive conclusions for television ads by organized groups render a mixed verdict. On the one hand, our findings suggest that issue ads can affect public opinion and can teach voters information. However, our results also suggest that not all ads work and that, given decay, groups must continue running the ads that do work as a controversy is ongoing. This result stands in contrast to prior experimental work finding that a single door-to-door conversation can produce long-term opinion change on these issues (Kalla and Broockman Reference Kalla and Broockman2020). If an organized group seeks to durably change attitudes, television advertising may not produce effects as large or durable; however, given the low per-person cost of TV ads, our confidence intervals are too wide to form confident conclusions about the relative cost effectiveness of TV advertising and personal contact.
Theoretically, placing our findings within the receive-accept-sample framework of Zaller (Reference Zaller1992), we found clear evidence that voters received the ads (as they did recall them). However, voters appeared to largely (although not entirely) reject accepting their messages, as their opinions usually did not durably change. Last, however, similar to Gerber et al. (Reference Gerber, Gimpel, Green and Shaw2011), the transitory effect of the ads on opinion suggests an interpretation of the effects we do find as representing priming, whereby when voters sampled relevant considerations to form opinions, the ads led them to be more likely to sample considerations consistent with the ads if they had seen the ad very recently. By contrast, these evanescent effects are inconsistent with models of on-line processing; rather than finding voters’ opinions changed even after they forgot the content of the ads, we found that voters remembered the ads even after their opinions returned to baseline. To be effective, ads may need to better reduce voters’ resistance to accepting their messages, perhaps with more credible sources or with more ideologically congruent arguments.
On a practical level, in close political battles, the small and short-lived effects we found might be consequential. However, our results also cast doubt on claims that organized groups’ ads have outsized influences on public opinion. It appears more apt to say that groups can have some influence on public opinion but their success is not guaranteed.
We hasten to note several limitations. First, this study examined the effect of organized group television advertising in isolation. During a full-fledged lobbying campaign, organized group outside lobbying might involve additional tactics and voters may pay more attention to any advertisements they do see, producing different effects. At the same time, our study participants were exposed to a large volume of advertising, suggesting that an artificially “low dose” of the ads is not to blame.
Second, this study only included four issue advertisements on two salient issues, both of which were advocating for policy change. It is possible that different advertisements on different issues or advertising attempting to block policy change may have different effects.Footnote 5 For instance, perhaps advertising might have effects on increasing issue salience on less well-known issues (e.g., Cooper and Nownes Reference Cooper and Nownes2004). With this said, the null findings we observe cannot be readily explained by the issues included in the experiment. Other research using similar designs has found that conversations can have large and long-lasting effects on LGBTQ and immigration attitudes (e.g., Kalla and Broockman Reference Kalla and Broockman2020).
Supplementary Materials
To view supplementary material for this article, please visit http://doi.org/10.1017/S0003055421001349.
Data Availability Statement
Research documentation and data that support the findings of this study are openly available at the Amercian Political Science Review Dataverse: https://doi.org/10.7910/DVN/VELEMB.
Acknowledgments
We thank Sarah Anzia, James Druckman, Martin Gilens, Donald Green, Seth Hill, Greg Huber, Ken Kollman, Eleanor Powell, Theda Skocpol, Lynn Vavreck, and Chris Warshaw for their helpful feedback and Akhil Rajan for his work as a research assistant. Programmatic support was provided by Kimberly Serrano of the California Immigrant Policy Center, the Movement Advancement Project, 76 Words, Amy Levin of Benenson Strategy Group, and Bobby Clark. We thank OpenLabs for implementing the advertising. This research was made possible thanks to a group of foundations supporting the Immigration Strategic Messaging Project. All remaining errors are our own.
Funding Statement
This research was funded by the Immigration Strategic Messaging Project.
Conflict of Interest
The authors declare no ethical issues or conflicts of interest in this research.
Ethical Standards
The authors declare the human subjects research in this article was reviewed by the Yale University Human Subjects Committee and determined to be exempt from review. The authors affirm that this article adheres to the APSA’s Principles and Guidance on Human Subject Research.
Comments
No Comments have been published for this article.