In October of 2016, President Obama claimed (and Democratic presidential nominee Hillary Clinton tweeted) that “civility is on the ballot.” Concern over political civility was widespread during the 2016 US presidential election, and many felt that the internet and social media (which Republican presidential nominee Donald Trump employed enthusiastically) were to blame.
Concern over incivility in contemporary political discourse can be traced back at least to the rise of cable news and its personalistic, outraged style (Mutz, Reference Mutz2015; Berry and Sobieraj, Reference Berry and Sobieraj2013). Indeed, concern about civil discourse accompanies any technological advance that lowers the cost of information production and distribution. The invention of the printing press led to elite concern about civil discourse during the time of the Reformation (Bejan, Reference Bejan2017).
Modern technological changes are taking place in the context of increased partisan animosity. Often called “affective polarization,” this animosity reflects a growing distrust and lack of respect among Democrats and Republicans (Iyengar, Sood, and Lelkes, Reference Iyengar, Sood and Lelkes2012). This phenomenon is directly related to civility, which Mutz (Reference Mutz2015) says is “a means of demonstrating mutual respect” (Reference Mutz2015, p7). Incivility is more than impoliteness: it is indicative of a disregard for the act of deliberation. Internet technologies may or may not be driving affective polarization, but they do at a minimum allow for the lack of mutual respect to manifest itself in uncivil online discourse.
Online communication lacks the evolved social and emotional feedback mechanisms that make it difficult to be uncivil in a real-world setting, and it affords physical distance and (sometimes) anonymity, decreasing the effectiveness of social sanctioning (Frijda, Reference Frijda1988). These technological affordances, in a context replete with bad actors intent on sowing discord for fun (Phillips, Reference Phillips2015) or geopolitical advantage (Chen, Reference Chen2015), have degraded norms of civil discourse online.
The implications of these changing norms are serious. Early enthusiasm about the capacity of the internet to democratize political discourse may have been premature (Hindman, Reference Chen2008), but the affordances of today’s ubiquitous, easy-to-use and social internet have caught up with the hype: in Reference Hindman2008, only 25% of US adults were on a social network, but during the 2016 US presidential campaign, that number was 68% (Greenwood, Perrin, and Duggan, Reference Greenwood, Perrin and Duggan2016).
However, survey evidence suggests that internet users do not feel that their interactions are civil or productive – “64% say their online encounters with people on the opposite side of the political spectrum leave them feeling as if they have even less in common than they thought” (Duggan and Smith, Reference Duggan and Smith2016).
I conducted an experiment that evaluates different strategies for promoting civil political discourse during the 2016 US presidential election. Using the method developed by Munger (Reference Munger2017), I used Twitter accounts that I controlled to sanction users engaged in uncivil discussions. In contrast to lab experiments conducted on a convenience sample in a short time frame, this approach allowed me to measure the effectiveness of sanctioning on a sample of frequently uncivil partisans in a realistic setting and in a continuous and unbounded time frame.Footnote 1
By manipulating the partisan identity of my “bots,”Footnote 2 I test the differential effects of sanctioning on Republicans and Democrats. By varying the language I tweeted at subjects, I test hypotheses about the relative effectiveness of two kinds of moral suasion and include a non-moral message that simply reminds subjects that what they are tweeting is public.Footnote 3
I found evidence of significant changes in subjects’ behavior, but the treatment effects were not what I theorized. Although one moral message caused a significant reduction in subject incivility in the first week after treatment and the other did not, these two treatment effects were not significantly different from each other. Furthermore, the theorized differential treatment effect on Democrat and Republican subjects was not observed, although the limited sample size means that this is evidence only against very large treatment effect heterogeneities.
Subject anonymity significantly moderated treatment effects, in the expected direction: more anonymous subjects (defined as the amount of identifying information provided on their Twitter accounts) were less likely to respond to the treatment. This trend was only significant in the 1-day time frame.
I also theorized that both moral treatments would be more effective at reducing incivility than a non-moral message that reminded users that what they were saying was public. I found that the moral messages caused a significant reduction and that this effect was significantly larger than the effect of the non-moral message.
These findings demonstrate that various different forms of moral suasion can be effective in promoting a more civil political discourse on Twitter, above and beyond the effect of merely calling attention to the subjects’ behavior. This moral suasion may only be effective on a subset of users; anonymous users (those more likely to be trolls) were unresponsive to moral suasion, and may even have been encouraged by being told that they were violating norms of political civility. Efforts to promote online civility should be sure to target the right people and use the most appropriate rhetorical strategy to maximize their efficacy.
Experimentally Reducing Political Incivility
The first step in performing this experiment was finding conversations that were uncivil, between out-partisans, and about politics. I thus used streamR, an R package developed to scrape the streaming Twitter Application Programming Interface (API) (Barberá, Reference Barberá2014), to find tweets mentioning either “@realDonaldTrump” or “@HillaryClinton” – the Twitter accounts of the two major party candidates in the 2016 US presidential election. I then dropped any tweets that were not directed at another user who was not either Trump or Clinton. In this way, I found a sample of tweets from non-elites that were concerned with the “issues” most likely to inspire political incivility in October 2016: Trump and Clinton. In order to filter through the hundreds of thousands of tweets every hour that fit these criteria, I used a machine-learning classifier designed to detect aggression, assigning an “aggression score” to each tweet I had scraped, then manually evaluated the top 10% most aggressive tweets per batch. Details of this measure are in the following section.
The first step was to check whether the uncivil language was directed at an account that appeared to be a member of the opposite political persuasion. Many of the potential subjects I found this way were tweeting at elites – either people verified on Twitter, journalists, or campaign operatives – and I excluded them.
When performing the manual inspection of the potential subject’s profile, I excluded users who appeared to be minors or who were not tweeting in English. I also checked to ensure that the subject’s profile was at least 2 months old; Twitter does ban some user accounts for harassment or other violations of their Terms of Service, so a very new account is likely to have been started by someone who had previously been banned.
For a visual overview of this selection process, see Figure 2. In this way, I found uncivil tweets from a non-elite to another non-elite with whom they disagreed politically. For an example, see Figure 1. @realDonaldTrump tweeted something, then Parker tweeted “you already lost” at Trump. Ty then responded to Parker (but because of how Twitter works, Ty’s tweet also “mentions” @realDonaldTrump) with an uncivil comment. Ty is the subject I included in the experiment, and because he was being uncivil to someone criticizing Trump, I coded Ty as a Trump supporter.
Figure 1 Finding non-elite incivility.
Figure 2 The decision process by which potential subjects were discovered, vetted and ultimately included or excluded. The right-hand column gives an example of the number of potential subjects excluded at each step in the process.
The sample in this experiment is thus not a “representative sample” of the general population, or even of Twitter users. The group of people encountered by following the steps listed in Figure 2 are, however, precisely the type of Twitter users who might be able to deliberate with and learn from their political opponents, if they were to do so in a civil fashion.
In the time since I performed this experiment, the prevalence of bots and Russian-backed trolls on Twitter has been revealed. Increasingly sophisticated “social spam bots” have been developed that are difficult for human coders to detect (Cresci et al., Reference Cresci, Di Pietro, Petrocchi, Spognardi and Tesconi2017). It is eminently possible that some of the subjects in my sample were undetected spam bots or Russian-backed trolls. Furthermore, based on the evidence showing that the majority of Russian bots were posing as Trump supporters, there may have been more of them in the Republican sample of my experiment. However, because my treatments were randomly assigned, there is no reason to think that the prevalence of illegitimate subjects varied across treatment conditions, so they should not pose a problem for inference.
I also recorded each subject’s Anonymity Score during the subject discovery process. The Anonymity Score ranged from 0 (least anonymous, full name and picture) to 2 (most anonymous, no identifying information). Ty, from Figure 0, was coded as a 1 – he chose to display what could plausibly be his full name.
My aim was to convince subjects that they were being sanctioned by a real person, so I made my bots look as real as possible. After I tweeted at a subject, they were likely to have received a “notification” from Twitter.Footnote 4 It is uncommon to be tweeted at by a stranger, but not extremely so, and especially not among a subject pool who are tweeting uncivil things at out-partisans. As a result, they were likely to click on my bots’ profile; if they did, they would see something very like Figure 3.
Figure 3 (a) Example Bot–Clinton Condition (b) Example Bot–Republican Condition.
Neil, in panel (a), was a bot who appeared to be pro-Clinton. I created four bots; the other three were pro-Democrats, pro-Trump, and pro-Republicans (see Todd, in panel (b)). To manipulate these identities, I changed the large banner in the middle of the profile, the small logo in the bottom right of the bots’ profile pictures, and the “bio” field below their username (eg “Hillary 2016!”; “Republicans 2016!”). The four bots were otherwise identical. All of the bots appeared to be white men, keeping the race/gender aspect of the treatment constant.
I took other steps in order to maximize verisimilitude. Most importantly, I ensured that all of the bots had a reasonably high number of followers. Munger (Reference Munger2017) varied the number of followers that sanctioning bots had and found that bots with few followers had very little effect. Based on this finding, I purchased 900 followers for each.
I created each bot in January 2015, giving the impression that they were long-time users. To further increase the perception that the bot was a real person, I tweeted dozens of innocuous observations and retweeted random (non-political) stories from the accounts the bots followed. There were two subject pools: people who were uncivil to people critical of Trump (“Republicans”) and people who were uncivil to people critical of Clinton (“Democrats”). Within each of these pools, each subject was randomly assigned one of three messages (“Feelings,” “Rules,” or “Public”) sent by one of two bots (pro-candidate or pro-party). There were initially 118 subjects in the “Republicans” pool, 104 subjects in the “Democrats” pool, and another 108 in the control group, to whom I sent no tweets.Footnote 5
The primary variation in the treatments was in the language of the message sent to the subjects. The aim was to convince subjects to change their behavior. One approach, employed in Munger (Reference Munger2017), is in-group social norm promotion: to cause subjects to update their beliefs about normative behavior for someone sharing their social identity. Munger found that sanctioning from bots that shared a social identity with the subject was more effective in changing their behavior than bots with a different social identity. To build on this finding, I held in-group social identity (in this case, partisanship) constant in the current study.
By varying the language of the in-group sanctioning, I tested the possibility of moral suasion. I based my approach on the moral intuitionist model proposed by Haidt (Reference Haidt2001), which argues that moral emotion is antecedent to moral reasoning. Extending the theory, (Haidt, Reference Haidt2012) argues that a necessary component for moral suasion is convincing your interlocutor that you are sympathetic and understanding. If the two of you share the same fundamental moral intuitions, you can reasonably discuss specific implications of those foundations, but if not, attempts to change their mind are likely to be interpreted as attacks on their worldview and to be met with resistance.
I designed two different treatments. The first was designed to appeal to the Care foundation, and thus to have some effect on Republicans but a much larger effect on DemocratsFootnote 6:
@[subject] You shouldn’t use language like that. [Republicans/Democrats] need to remember that our opponents are real people, with real feelings.
The other treatment appealed to the Authority foundation. My expectation was that it should have an effect on Republicans but not on Democrats:
@[subject] You shouldn’t use language like that. [Republicans/Democrats] need to behave according to the proper rules of political civility.
In addition to these moral foundations treatments, I included a non-moral “public” treatment. My intention was to use a message that would serve to remind subjects that their uncivil tweets were public, and my hypothesis was that this treatment would decrease the subjects’ use of incivility, but that the effect would be smaller than the moral treatments’. To that end, I designed a message that emphasized the subject’s visibility:
@[subject] Remember that everything you post here is public. Everyone can see that you tweeted this.
Hypothesis 1 The reduction in incivility caused by the Care condition will be larger for Democrats than for Republicans. There should be a reduction in incivility caused by the Authority condition for Republicans, but not for Democrats. There should be a reduction in incivility cased by the Public condition, but it should be smaller than the other effects.
Some subjects are more heavily invested in their online identities than are others. Twitter allows individuals to decide how much personal information to divulge, so while some users are completely anonymous, others include their full name, picture, and biography. Users who are more invested in their online identities are more likely to change their behavior in response to sanctioning, while anonymous users are unlikely to do so.Footnote 7
Hypothesis 2 The reduction in incivility caused by the treatments will positively covary with the subject’s Anonymity Score.
Results
The behavior targeted in this experiment is partisan incivility targeted at other Twitter users. To capture this behavior, I scraped each subject’s Twitter history before and after the treatment and restricted the sample to the tweets that were “@-replies”: tweets directed at another user. After removing the 18 users for whom I could not collect enough pre- or post-treatment tweets (see Appendix A for a full discussion), I used the model trained by Wulczyn, Thain, and Dixon (Reference Wulczyn, Thain and Dixon2017) to assign an “aggression score” (between 0 and 1) to each of these 367 thousand tweets. This measure was skewed toward the lower end of the distribution, so I selected all tweets above the 70th percentile aggression score and coded them as incivil.Footnote 8
I selected the 70th percentile based on the empirical distribution of aggression scores and the concordance with the validation data discussed below. See Appendix B for the empirical distribution. This exact cutoff was not specified in my pre-analysis plan, but the fact that I would be using this model was pre-registered.
Aggression is far from the only measure of incivility that scholars have proposed. Note, however, that the current model measures aggression in a set of subjects who have been aggressive to non-elite members of the out-party, making it similar to the definition of incivility used by Theocharis et al. (Reference Theocharis, Barberá, Fazekas and Popa2015). Perhaps the most similar operationalization is the “personal-level incivility” defined by Muddiman (Reference Muddiman2017), who find that this form of incivility is evaluated by subjects on Mechanical Turk to be more uncivil than other conceptions of incivility.
This result mirrors my own validation of the Wikipedia model. I had a random sample of 1,000 subject tweets labeled as either “civil” or “uncivil” by crowdworkers on Mechanical Turk. The model correctly predicted the human-generated labels 81.2% of the time while ensuring a balance between type 1 and type 2 classification error (for a more extensive discussion, see Appendix C).
The aggression model is theoretically consonant with others in the literature, but equally importantly (in my view), it was specified in my pre-registration of this research. This pre-registration is particularly important when using measures derived from text. Text data are extremely high dimensional, so the development of measures ex post allows researchers (often unknowingly) to select the measure out of millions of potential measures that best supports their hypothesis.
To control for each subject’s pre-treatment behavior, I calculated their rate of uncivil tweeting in the 3 months before the experiment. This measure was included as a covariate in all of the following analysis.
The data take the form of overdispersed count data: the variables that record the number of uncivil tweets sent by each user are bounded by zero and vary widely between highly active and normal users. Table 1 reports these distributions. To account for this high variance, I take the log of the uncivil tweet count variables in the following results.Footnote 9
Table 1 Distribution of Incivil Subject Tweets, Pre-, and Post-Treatment
The experimental results on the full sample with all treatments pooled are displayed in Figure 4. In all of the analysis that follows, the dependent variable is the (log of the) number of uncivil tweets the subject sent in the specified time period. Each of the four results is for a given non-overlapping time period: the first result is for the first day after treatment, the second result for days 2–7, the third for days 8–14, and the fourth for days 15–28.
Figure 4 Pooled treatment effects on the entire sample, controlling for the log of the number of pre-treatment uncivil tweets sent by each subject. Lines represent 95% confidence intervals.
In none of the these four time periods is there a statistically significant treatment effect with all three treatments pooled. To test the second part of Hypothesis 1 (that the effect of the two moral treatments would be larger than the Public treatment), Figure 2 pools the three treatments into these two categories. In Week 1, there is a statistically significant effect of the two moral treatments, while the effect of the Public treatment is actually slightly positive. These treatment effects are statistically significantly different from each other (p < .02).
Although not an explicit hypothesis, it is plausible that treatment effects should decay over time. This does not seem to be the case in Figure 5: there are no significant treatment effects in the first day after treatment, but there are in the first week after treatment. This is to some extent an issue of data size: the first 24-hour time period is simply too noisy. The results in Appendix D support this conclusion: if the analysis is run with the higher threshold for classifying tweets as uncivil (thus decreasing the number of tweets in the analysis), the point estimate of the effect in Day 1 becomes significant for the moral treatments.
Figure 5 Pooled treatment effects on the entire sample.
Figure 6 further disaggregates the treatment effects. In the main Week 1 time period, there is a significant effect of the Care treatment, but not the Authority treatment. These treatment effects are not significantly different from each other, however.
Figure 6 Treatment effects on the entire sample.
To test the first portion of Hypothesis 1, Figure 7 breaks down Figure 6 by the partisanship of the subjects. The top panel displays the results for Republicans, and the bottom panel for Democrats.
Figure 7 Treatment effects divided by subject partisanship.
These results do not support the partisan portion of Hypothesis 1.
To test Hypothesis 2, I re-ran the analysis in Figure 6 with an interaction term between subject anonymity (on the three-point scale, where 0 means they provided a full bio and 2 means they were fully anonymous). Figure 8 reports the results for the Day 1 (top panel) and Week 1 (bottom panel) time periods.
Figure 8 Treatment effects divided by subject anonymity.
In the Day 1 time period, subject anonymity behaves as expected for the two moral treatments: less anonymous subjects change their behavior more by sending fewer uncivil tweets. This interaction effect is statistically significant for the Care treatment at p < .05, but not for the Authority treatment (p=.11).
This trend does not obtain in the Week 1 time period. There is no evidence of any interaction effect for the Authority or Public treatment conditions, and the evidence for such an effect with the Care condition is very weak. There were no significant treatment effects for the later time periods, and similarly no evidence of heterogeneous treatment effects (results not shown).
One possible explanation for the lack of the expected effect on Democrat subjects is that this group was more heterogeneous. I explore this possibility in Appendix H. I find that some of the accounts coded as Democrats were actually anti-Trump Republicans, who were understandably unmoved by an appeal to the Democrat in-group.
Conclusion
The 2016 US presidential election took place during a time of rapidly changing norms of political civility. Although civility had been associated with conservatism, especially during the latter half of the 20th century, the campaign of Donald Trump embraced incivility while Hillary Clinton explicitly positioned herself as the candidate of civility.
The idiosyncrasies of this election may explain my failure to find support for the first half of Hypothesis 1: there was little difference between sanctioning language designed to appeal to Democrats’ and Republicans’ moral sense of Care or Authority. The small sample size of the experiment does not allow me to rule out the existence of moderate differences in treatment effects by partisanship, but I do find evidence against massively effective interventions targeted at subjects’ specific moral framework.
The post hoc explanation for why Trump supporters did find the argument that they should “behave according to the proper rules of political civility” compelling is straightforward. More subtly, the minimal response from Democrats to the Care treatment may be explained by the tweets they sent to my bots in response to being sanctioned. In several cases, Democrats told my bots something like “these other people are Trump supporters, so I don’t care about their feelings”; no Republicans expressed a similar sentiment. This is in keeping with the theory developed in Haidt (Reference Haidt2012): the morality of Democrats in the US is based largely on care for specific historically disadvantaged groups, a category which does not include Trump supporters.
However, I do find support for the second half of Hypothesis 1: overall, the two moral messages caused a significant reduction in incivility in the first week after treatment, and this estimated effect was significantly larger than that of the non-moral treatment. In short, despite the oddities of the election, my intervention successfully reduced partisan incivility.
Importantly, this reduction was not observed in the models that examine only the first day after treatment. There are two explanations for this: first, this 24-hour time period is simply too noisy because the sample of tweets is too small (see Appendix D for a discussion of this point). Second, this is the period in which individuals responded directly to the treatment message. For a subset of subjects, this entailed a short-term increase in incivility that counterbalanced the reduction in other subjects in this time period. Had my research design been restricted to this time period, my inferences would have been very different. Future research in this area should aim to measure effect persistence.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/XPS.2020.14
