While academic studies of voter mobilization have taught us much about what drives turnout,Footnote ¹ we know little about the effect that get-out-the-vote (GOTV) campaigns have on election outcomes. Experimental work has shown that there are heterogeneous treatment effects – in other words, that encouragements have different turnout effects on different people.Footnote ² However, there is little evidence from turnout experiments about whether mobilization efforts affect the outcome of the vote. One reason the GOTV literature primarily focuses on turnout is data availability. Individual vote choice is difficult to observe directly due to the secret ballot, while administrative turnout records can be obtained in many places. The few previous studies of GOTV on vote choice look at partisan mobilization and rely on incomplete self-reported or aggregate vote choice.Footnote ³ As a result, we do not know whether GOTV campaigns actually mobilize people who vote differently than those who would otherwise have voted.

Several studies have looked at the relationship between turnout and policy outcomes at the national level, and consistently find that higher turnout is associated with higher levels of redistribution.Footnote ⁴ However, these studies rely on observational data and cannot assign higher turnout experimentally.

This study has two main outcomes of interest. First, we examine the vote encouragement treatment effects on online turnout in a participatory budgeting (PB) vote. This part of the study is similar to much of the established GOTV literature, in which the outcome of interest is whether an eligible voter turns out to vote on election day. Secondly, we study the effect of GOTV treatments on the actual vote using administrative records of vote choice. To our knowledge, this has never been studied before using real vote data. The nature of the PB vote in our case allows us to study both the average cost of proposals selected by voters as well as specific sectoral (health, education, environment etc.) preferences.

To test whether increased turnout from voter mobilization affects the popular vote, we conducted a randomized controlled trial during the June 2014 participatory budgeting vote in Rio Grande do Sul (RS) in southern Brazil.Footnote ⁵ Note that this is not a regular election in which votes are cast for candidates and parties. In a PB vote, voters are asked to choose between specific public investment projects that they can indicate their support for. We devised a procedure in close collaboration with the implementing government agency that retained an indicator of individual-level treatment assignment with the actual vote choice record. The procedure anonymizes all other individual attributes, thus preserving the secrecy of the ballot.

The list of experimental subjects consists of 43,384 voters who took part in the annual online PB vote in the past two years. The list includes only individual voters who voluntarily provided both an email address and a phone number when voting in the past. The population we study appears to have a higher propensity to vote than the rest of the online voting population. Examining historical online PB vote data, we find that 22.4 per cent of voters for whom we have an email address took part in the online vote in both 2012 and 2013, compared to 18.2 per cent of all online voters.

We randomly assigned subjects, in equal proportions, to one of four groups: control, informational, public benefit of voting, and private benefit (lottery reward) using a simple random allocation scheme.Footnote ⁶ All of the messages were non-partisan in nature and were focused on increasing turnout. The sender of the email was Gabinete Digital, the governor’s digital engagement unit. Figure 1 shows the email that subjects in the informational treatment group received in the morning on the first day of online voting (2 June). Each experimental subject was sent three emails: a voter registration email on 30 May, a message about voting starting on 2 June, and a message about it being the last chance to vote on 4 June. We also sent out a mobile phone text message (SMS) on 3 June.

Fig. 1 Informational treatment message on election day 1 on June 2 Note: more details about the treatments are available in the Appendix.

The content of the message was the same throughout the process for each treatment condition, while the exact wording of SMS messages varied due to space limitations. On 2 June, the first treatment group was sent an informational message stating basic information about this year’s online voting taking place from 2–4 June. The message subject line read ‘Today: Priorities Vote!’ and the body text contained a direct link to the voting site. The two other groups were also sent the informational message, but the public benefit motivation condition was also assigned the following message: ‘This is your chance to make government work on your priorities.’ In addition, the subject line for the public benefit condition included the words: ‘Make a difference.’ The private benefit condition instead received the following message: ‘If you vote in via Citizen Login you can win two extra tickets for the raffle at R$ 1 million occurring in March 2015.’

In this GOTV experiment, we study the final stage of an annual three-month long budget consultation process that culminates in citizens voting on their priorities for a pre-screened set of public investment projects and policy sectors. In the PB vote in RS the ballot is split into two sections: a list of up to 20 possible public investment projects with specific costs attached to each of them, of which a voter can pick up to four projects, and a list of five possible regional sectoral (health, education, etc.) priorities, of which a voter can choose up to two options. The items on the ballot vary across the twenty-eight PB areas depending on the projects suggested within the PB meetings held prior to voting day, and a few of the regions do not list costs on the ballot.Footnote ⁷ However, on all ballots, each of these items is coded into one of the same fourteen possible thematic areas, allowing us to compare vote choices across these electoral areas.

Because the secrecy of the vote is a key tenet of electoral integrity, the GOTV literature has had to rely on self-reported or aggregate-level voting when studying vote choice. To ensure that individual voters could not be identified in our study, the state agency that administered the online vote (PROCERGS) merged a vector with the treatment assignment into the vote records data prior to decrypting an individual’s vote choice. After the vote verification process was completed (checking for duplicate voters, voters voting in the wrong area, etc.), the vote record stripped individual identifiers, while retaining the treatment assignment vector. Only at this point were the votes decrypted. The vote records could then be transferred to the research team safely, without compromising individuals’ privacy. At no point in the voting process were decrypted votes and personally identifiable information kept on the same server. This approach is analogous to voters being given a different colored ballot when their identification is checked at the polling station depending on their treatment group. The analysis is then conducted solely in terms of differences in votes cast using different colored ballots. Since the number of experimental groups is extremely small compared to the number of voters, the different ballots do not meaningfully increase the post hoc identifiability of voters.

Prior to treatment assignment, we submitted a pre-analysis plan to Experiments in Governance and Politics. As specified in the plan, we estimate the intent-to-treat (ITT) effect by using a difference-in-means estimator. Compliance with treatment, defined as the recipient having read the subject line or the body text, is difficult to determine since we do not know whether the mail ended up in a recipient’s junk folder, but the bounce rate was a very low 1.6 per cent. With non-compliance this low, effectively the estimand is the Average Treatment Effect (ATE). In the linear model we include dummy variables for each of the twenty-eight regions (COREDES) since the ballot varies across regions. We address the multiple comparisons problem using Benjamini and Hochberg’s method for adjusting p-valuesFootnote ⁸ to control the false discovery rate (FDR) for each dependent variable.Footnote ⁹

The first substantive finding is that different GOTV treatments have a turnout-increasing effect (see Figure 2). Taken together, voters assigned to receive any of the GOTV messages were 4.7 percentage points more likely to vote in the online poll (ITT, difference in means linear estimation, p<0.001, see Table 1). Turnout in the control group was 22.6 per cent. Each of the separate mobilization treatments shows significant differences from the control group that received no email. The content of the mobilization messages does not matter much, a finding that is broadly consistent with the literature.Footnote ¹⁰ However, the private benefit (lottery) treatment did perform significantly worse than the informational treatment (−1.6 percentage points, difference in means linear estimation, FDR-adjusted p=0.01).

Fig. 2 Online turnout in control, pooled treatment group and the three treatments separately Note: the three treatments are informational, public benefit and private benefit. 95 per cent confidence intervals are displayed.

Table 1 Treatment Effects on Turnout – Main Model and Comparisons Across Different Treatment Subsets

The turnout effect we document here is substantially larger than that found in most previous GOTV studies,Footnote ¹¹ particularly those focusing on the effect of new technologies, like email and SMS.Footnote ¹² While we cannot test the reasons for this performance directly, it is likely due to repeated messages and multi-mode contact, and the ability to vote by clicking on a link in the message. In addition, while this is a low-salience election, almost a quarter of the subjects targeted were habitual voters, for whom large effects have been found.Footnote ¹³ The high performance of the informational message suggests that one of the primary ways in which the GOTV message is working is simply by informing voters that there is an election. The lower performance of the private benefit message could be for a number of reasons. For example, a private benefit may undermine public benefit motivation,Footnote ¹⁴ it may be more likely to be caught by spam filters, or voters may simply be less trusting of such a message. Regardless of the reason, these results suggest that large lottery rewards are not an especially effective way of mobilizing voters.

In the second part of the study, we examined unique administrative vote records. Previous studies have shown that non-voters differ from voters,Footnote ¹⁵ which implies that additional voters brought in by GOTV efforts will have an effect on election outcomes by altering the preferences represented. We test this implication directly by examining vote choice across the different treatment groups. As specified in the pre-analysis plan, we first test whether our treatments affect the average cost of proposals selected by voters across different groups. Each proposal was listed on the ballot with a Brazilian Real (R$) cost attached to it. While the total amount of money spent is not directly affected by the choices a voter makes, we view a lower total cost on a ballot as reflecting a voter’s preference for lower spending. We operationalize cost as the average cost of the items a voter chose on the ballot, expressed as a proportion of the maximum cost they could have picked on their ballot. In the control group that number was 65.1 per cent. As can be seen from Table 2, the total costs do not differ significantly between the control and treatment groups (−0.2 percentage points, difference in means linear estimation, FDR-adjusted p=0.79). Similarly, none of the individual treatments shows differences from the control group or each other. These results show that encouraged voters do not differ from natural voters in terms of their preferences for higher- or lower-costing public investments.

Table 2 Treatment Effects on Cost

Note: main model and comparisons across different treatment subsets. Average cost of options chosen on a respondent’s ballot as a percentage of the maximum cost they could have chosen.

In addition to the cost of proposals, we also consider whether encouraged and un-encouraged voters differ in their preferences for different public policy sectors. We compare the proportion of voters that voted for an item in the three most popular issue areas displayed on the second part of the ballot. In the 2014 PB vote, these were health, infrastructure and security. We also include an ‘other’ category for the proportion that voted for any of the other categories. We analyze only the votes on the second section of the ballot, where a voter can choose up to two priorities for their own region (Prioridades Regionais).

Table 3 shows the proportions that chose each area across the control and treatment conditions. As with the total cost, we find no differences between the control and pooled treatment groups in their preferences for health (FDR-adjusted p=0.98), infrastructure (FDR-adjusted p=0.98), security (FDR-adjusted p=0.98) or other spending (FDR-adjusted p=0.98). We also tested whether voters in the different treatment groups differed from each other if, for instance, voters encouraged to participate by a public benefit message might be more inclined to support health projects. In all cases, we found no differences in sectoral preferences according to treatment condition. Taken together, these results provide no evidence that our GOTV treatments had any impact on the outcome of the online vote.

Table 3 Treatment Effects on Vote Choice

Note: main model and comparisons across different treatment subsets. The average proportion of voters choosing the top three sectoral categories: health, infrastructure or security, and an ‘other’ category combining all other categories.

Based on the size of the GOTV effect and the proportions of voters in the treatment and control groups, we can examine the implied size of the difference in preferences between regular (voters who would have voted anyway) and encouraged (voters who only voted because of the GOTV treatment) voters (although none of these differences is close to significant, as we discussed above). Applying the method outlined by Fowler,Footnote ¹⁶ the largest implied difference is voting for ‘other’, which we estimate as 4.8 percentage points less likely among encouraged voters (see Table 4). However, all of these differences have confidence intervals encompassing zero. In the Appendix, we show a power analysis that suggests that we would expect to be able to detect a 19-percentage-point difference 83 per cent of the time. This means that we can be reasonably confident that there are no large differences in sectoral preferences between regular and encouraged voters. We cannot, for instance, rule out the possibility that differences in the size seen here of 1.8–4.8 percentage points might remain and become significant with a larger sample size.

Table 4 Estimated Differences in Sectoral Preference Between Regular and Encouraged Voters and Difference in Chosen Cost

Note: confidence intervals estimated using bootstrapping.

An important implication of our study is that email messaging can be a powerful tool to mobilize people to take part in a PB vote, especially if the type of engagement allows for online modes of participation. This is particularly vital in the context of Brazil, which has experienced a wave of protests related to, among other grievances, government spending. Democratic innovations that allow citizens to participate directly in budgeting are seen as an important way of increasing citizen engagement and government legitimacy, and high rates of participation are considered very important for the credibility and sustainability of these initiatives.Footnote ¹⁷

Future research should focus on replicating these results in other contexts. As the world moves toward more online voting, analyzing vote choice may become more feasible in more cases. However, we would encourage researchers to think carefully about voter privacy. The focus in this study, as in most of the literature, was on non-partisan voter mobilization. Needless to say, we still expect that partisan GOTV campaigns targeting their own core supporters or particular demographic groups would shift the election outcome by increasing their own supporters’ turnout. However, this would be the result of differential targeting rather than heterogeneous treatment effects.

This research calls into question the assumption that changing the pool of voters by increasing turnout will inevitably affect the distribution of votes cast. Unlike observational studies, we do not find that higher turnout affects voters’ policy preferences, and we can be confident that the new voters brought in via the GOTV campaign do not have drastically different sectoral preferences. There are a number of possible reasons for these divergent findings. One possibility is that another factor explains both higher turnout and redistribution across countries. However, there are also important contextual differences that could explain our findings. In a low-salience and non-partisan election, the voters who are mobilized through GOTV may be sufficiently similar to those who were already turning out to vote that voters’ choices are not distinguishable. It is also possible that endogenous increases in turnout are more likely to bring in voters who vote differently to existing voters than increases in turnout generated through untargeted GOTV. Finally, the choices in a PB vote may be less correlated with turnout than in partisan elections, which have more established signals for voters to use. Nonetheless, it is striking that the experimental evidence of turnout’s effect on policy choices comes to such a different conclusion than previous observational evidence.

While it might initially seem that our results imply that there is no value in using non-partisan GOTV to increase turnout, this assumes that the only value of voting is to change the outcome of an election. Many arguments have been made for increasing turnout, including legitimizing the political system, decreasing the alienation of the populaceFootnote ¹⁸ and even as a form of education.Footnote ¹⁹ The results of this study should encourage non-partisan GOTV campaigns precisely because they are unlikely to greatly affect the results.