Muslims are one of the most stigmatized groups in Western liberal democracies, with half of all US adults believing that Islam is not part of mainstream American society, and over 40% feeling that Islam encourages violence more than other faiths (Pew Research Center and Lipka Reference Lipka2017). Although several factors undoubtedly influence these perceptions, a 2007 survey revealed that more Americans get their information about Muslims from the media than from any other single source and that Americans' attitudes are markedly more negative when they learn about Muslims from the media rather than through interpersonal relationships or education (Pew Research Center 2007).Footnote 1 Lajevardi has argued that the limited first-person contact most Americans have with Muslims means that the media remain highly influential in shaping attitudes toward this group (Lajevardi Reference Lajevardi2020, 109–10).
In spite of decades of scholarly research into media coverage of Muslims (see Ahmed and Matthes Reference Ahmed and Matthes2017), its precise relationship to public attitudes is only beginning to come into focus. Recent scholarship has demonstrated that coverage of Muslims is substantially more negative than coverage of ethno-religious groups such as Catholics, Jews, and Hindus (Bleich and van der Veen Reference Bleich and van der Veen2021) or of ethno-racial groups such as Blacks, Latinos, and Asian Americans (Lajevardi Reference Lajevardi2021), that greater news exposure is associated with more negative attitudes toward Muslims (Shaver et al. Reference Shaver, Sibley, Osborne and Bulbulia2017), and that negative stories about Muslims are linked to greater support for civil restrictions of Muslim Americans and for military action in Muslim countries, as well as to increased opposition to Muslim refugee resettlement, (Saleem et al. Reference Saleem, Yang and Ramasubramanian2016; Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Nassar Reference Nassar2020). Scholars have also shown that exposing viewers to stereotypic negative topics in broadcast news stories increases resentment toward Muslim Americans, while the results are mixed for exposure to counterstereotypic positive topics (Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Lajevardi Reference Lajevardi2020, 108–31, Reference Lajevardi2021). These are important findings, yet they have not established precisely how variations across a range of media tone shape individual attitudes.Footnote 2
We build on this work to deepen our understanding of how media coverage affects attitudes toward Muslims. We use experimental methods to examine the effect of article tone (more specifically, valence) on attitudes. We leverage a dataset that provides precise assessments of the valence of hundreds of thousands of newspaper articles about ethno-religious groups, all calibrated against a representative sample of articles drawn from 17 national and regional US newspapers over a 21-year period. We conduct a between-subjects online experiment in which participants read pairs of short negative, neutral, or positive articles about Muslims, and we assess the effect of article tone on attitudes about the group.
While our primary interest is in the effect of media tone on Muslims, we probe the possibility that attitudes toward Muslims may have distinct qualities (Oskooii et al. Reference Oskooii, Dana and Barreto2021) by conducting a parallel experimental treatment to examine attitudes toward Catholics. This is a novel approach given that a meta-analysis of studies of media coverage of Muslims reveals that they rarely include a group-comparative dimension (Ahmed and Matthes Reference Ahmed and Matthes2017). We chose Catholics because, in many significant ways, they are a “most different” American minority from Muslims. Whereas Muslims are viewed as a relatively new immigrant group, Catholics have been present in large numbers in the United States for well over a century. Muslims constitute approximately 1% of the US population, whereas Catholics make up over 20% (CIA 2020). Finally, Catholics receive broadly positive coverage within the American media compared to the significant negativity associated with Muslims (Bleich and van der Veen Reference Bleich and van der Veen2021). In short, Catholics are much less stigmatized than Muslims in the US context (Kalkan et al. Reference Kalkan, Layman and Uslaner2009). Yet our experimental results show similar effects of media valence across these two very different categories, which suggests that the dynamic we identify may apply not just to Muslims and Catholics but also to a wide variety of social groups.
Our analysis demonstrates that exposure to articles of different valences affects attitudes toward both groups. The effect is most consistent for respondents treated with Catholic articles, where there are distinct differences in attitudes that correspond to the tone of the articles read by subjects. The effect is also present with respect to Muslims, but tapers off when comparing neutral and positive articles. Our main finding is thus that the valence of news articles shapes attitudes. In addition, we show that while the effect of negative news about the two groups is similar, positive news increases attitudes much less for participants treated with news about Muslims. Our study thus uses actual newspaper articles and a novel experimental design to contribute to scholarship examining the effects of counterstereotypic or positive representations of Muslims (Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Lajevardi Reference Lajevardi2020, Reference Lajevardi2021; Williamson Reference Williamson2020).
To further probe the relationship between media tone and attitudes, we perform a series of robustness checks designed to test the limits of our first-order findings. As another novel contribution, we examine the effect of article valence on attitudes not only when measured by a feeling thermometer, but also when money is at stake. To our knowledge, studies that have tested media effects on public attitudes toward Muslims have not assessed whether respondents are willing to pay for their beliefs. We draw on the experimental technique of the dictator game (Forsythe et al. Reference Forsythe, Horowitz, Savin and Sefton1994) to identify whether the level of donations to Muslim or Catholic charities varies based on the valence of articles the respondent reads. This tests the robustness of the causal link by altering the incentives to truthfully report one's attitude. We find some evidence for the effect of tone on donative behavior, which is once again clearer for articles about Catholics than for those about Muslims.
We then explore a key additional dimension of the relationship between media tone and attitudes about social groups by examining anxiety as a potential mediator between treatment and outcomes. Scholars have long identified anxiety as an influential emotion that affects information gathering, processing, and attitudes (Mathews Reference Mathews1990; Brader et al. Reference Brader, Valentino and Suhay2008; Gadarian and Albertson Reference Gadarian and Albertson2014). We hypothesize that exposure to articles of different tones affects respondents' anxiety levels, and that these effects are passed through to the attitudes about each group. Our analysis demonstrates that anxiety does indeed function as a mediator between article valence and attitudes about each group. We thus show not only that article tone affects attitudes, but also pinpoint a mechanism through which it does so.
Our overarching goal is to understand whether, to what extent, and through which pathways attitudes toward Muslims and other social groups are influenced through everyday modes of communication such as the media. Our findings suggest that exposure to stimuli of different tones can have an immediate impact on key aspects of how readers view not only Muslims but also Catholics, which, in turn, may help us understand how repeated exposure to articles of different tones affects long-term views of groups in ways that structure social relations.
Theoretical Background
We focus on Muslims as a core category of analysis given their emergence over the past two decades as a racialized social outgroup of increasing visibility that faces significant stigmatization that scholars argue may be specific to that group (Oskooii et al. Reference Oskooii, Dana and Barreto2021; Lajevardi Reference Lajevardi2020). In American society, Kalkan, Layman, and Uslaner situate Muslims within two distinct “bands of others”: those like Latinos or African Americans with ethnic or racial characteristics, and those such as welfare recipients, illegal immigrants, or atheists that are defined by values and behaviors that “citizens in the mainstream…find troubling” (Kalkan et al. Reference Kalkan, Layman and Uslaner2009, 3). Muslims are consistently associated with a range of negative evaluations in surveys and tests of implicit biases (Park et al. Reference Park, Felix and Lee2007; Pew Research Center and Lipka Reference Lipka2017). There have also been numerous efforts over the past decade to demonstrate the presence of the specific trait of Islamophobia among respondents (Imhoff and Recker Reference Imhoff and Recker2012; Kunst et al. Reference Kunst, Sam and Ulleberg2013; Lee et al. Reference Lee, Reid, Short, Gibbons, Yeh and Campbell2013).
Reviewing decades of research on media portrayals of Muslims in the US and beyond, Ahmed and Matthes (Reference Ahmed and Matthes2017) find a great degree of negativity. While interest in representations of Muslims has spiked since the 9/11 attacks, observers such as Edward Said identified problematic media portrayals of Islam as early as the 1980s (Said Reference Said1997 [1981]). Over the past two decades, researchers using qualitative analyses involving close readings of handfuls of articles or quantitative overviews of hundreds or even hundreds of thousands of articles have highlighted the prevalence of stereotypical negative portrayals of Muslims in the US and international media (Abrahamian Reference Abrahamian2003; Trevino et al. Reference Trevino, Kanso and Alan Nelson2010; Powell Reference Powell2011; Baker et al. Reference Baker, Gabrielatos and McEnery2013; Terman Reference Terman2017).Footnote 3 In addition, survey research of over 16,500 New Zealanders tracked a correlation between reported hours of news consumption and higher levels of anger and lower levels of warmth toward Muslims (Shaver et al. Reference Shaver, Sibley, Osborne and Bulbulia2017). Experimental work by Saleem et al. (Reference Saleem, Yang and Ramasubramanian2016, Reference Saleem, Prot, Anderson and Lemieux2017), by Lajevardi (Reference Lajevardi2020, 121–25), and by Nassar (Reference Nassar2020) shows that exposure to negative media coverage of Muslims increases support among American respondents for civil restrictions for Muslim Americans, for restrictions on Muslim immigration and refugee resettlement, and for military action in Muslim countries.
To assess the strengths and limitations of existing scholarship, it is helpful to situate these studies in the literature that examines media effects on attitudes toward groups such as immigrants or minorities (Entman and Rojecki Reference Entman and Rojecki2001; Boomgaarden and Vliegenthart Reference Boomgaarden and Vliegenthart2009; van Klingeren et al. Reference van Klingeren, Boomgaarden, Vliegenthart and de Vreese2015; Eberl et al. Reference Eberl, Meltzer, Heidenreich, Herrero, Theorin, Lind, Berganza, Boomgaarden, Schemer and Strömbäck2018). Most of this research traces a relationship between media tone and attitudes by drawing on examples of “topical negativity,” such as coverage that links groups to stereotypes of poverty, crime, violence, or other forms of threat (Gilliam and Iyengar Reference Gilliam and Iyengar2000; Schemer Reference Schemer2012; Schlueter and Davidov Reference Schlueter and Davidov2013). While these demonstrate that topically negative coverage about marginalized groups is associated with negative attitudes, they have a number of drawbacks. So far there has been no systematic way to estimate how the degree of negativity influences attitudes. It has also been relatively uncommon for scholars to examine the effect of neutral or positive coverage on attitudes (though see Schemer (Reference Schemer2012), van Klingeren et al. (Reference van Klingeren, Boomgaarden, Vliegenthart and de Vreese2015) and Schmuck et al. (Reference Schmuck, Heiss and Matthes2020)). Moreover, because this research relies heavily on stereotypes as a marker of negativity, it is difficult to estimate the effect of article tone on less-marginalized groups to which negative stereotypes seldom apply.Footnote 4
Recent scholarship on Muslims has gone some distance toward addressing one of these key limitations. Both Saleem et al. (Reference Saleem, Prot, Anderson and Lemieux2017) and Lajevardi (Reference Lajevardi2020, 108–31, Reference Lajevardi2021) have compared the effects of negative and positive treatments on respondent attitudes when treated with video news clips or cable news transcripts. While Saleem et al. found that counterstereotypic treatments reduced perceptions of Muslims as aggressive when compared to stereotypic treatments (Saleem Reference Saleem, Prot, Anderson and Lemieux2017, 855–56), Lajevardi's experiments led to mixed results, leading her to conclude that positive coverage “does not appear to work in the same way” as negative coverage (Reference Lajevardi2020, 130).Footnote 5 Even given notable advances, these studies continue to rely on specific topics of coverage rather than on stories that are selected based on measurable valence, and they do not compare the effects of media tone on less-marginalized groups to assess the applicability of their findings across a wider spectrum of social groups.Footnote 6 They also each rely on television as a medium and come to mixed or inconclusive findings about the effect of positivity on attitudes, which invites further research on different media.
To address these limitations, we utilize a dataset of newspaper articles that provides precise estimates of the scale of article valence and examine the effect of exposure to negative, neutral, and positive articles about both a stigmatized and non-stigmatized group. Given the potential distinctiveness of responses to Muslims—especially the stickiness of negative attitudes toward Muslims (Lajevardi and Oskooii Reference Lajevardi and Oskooii2018) and the uncertainties surrounding the effects of positive representations (Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Williamson Reference Williamson2020; Lajevardi Reference Lajevardi2021)—we introduce Catholics as a comparison group. As noted above, Catholics are a non-majority religious group in American society that is arguably as different from Muslims as possible. Although not part of the Protestant majority, Catholics are Christian, they make up over 20% of the American population (CIA 2020), and they are seen as having longstanding roots in the United States. Kalkan, Layman, and Uslaner find that Catholics, although once viewed as outsiders, have “moved more or less into the American mainstream” and are now “accepted by mainstream society” largely due to increased contact and familiarity (Kalkan et al. Reference Kalkan, Layman and Uslaner2009, 3). This proposition is supported by generally positive attitudes toward Catholics in surveys, especially when compared to attitudes toward Muslims (Putnam et al. Reference Putnam, Sander and Campbell2006, Reference Putnam, Sander and Campbell2011; Pew Research Center 2014). While we expect the effects of our treatments to be more straightforward with regard to Catholic articles, we also anticipate they will hold with respect to Muslim articles. Our study thus allows us to explore the generalizability of media effects to a greater degree than has been possible so far.
We follow most research in our use of feeling thermometer ratings to gauge attitudes toward social groups (Putnam et al. Reference Putnam, Sander and Campbell2011; Pew Research Center 2014; Shaver et al. Reference Shaver, Sibley, Osborne and Bulbulia2017). This measure has long been applied to a wide variety of policies, political parties, and ethnic, racial, and religious groups (Nelson Reference Nelson2008). However, we expand upon this approach by introducing a supplementary measure of attitudes that involves monetary incentives. Our aim is to examine the robustness of attitudes when it is individually costly to exhibit either positive or negative sentiments toward social groups. Behavioral and experimental economists have long used dictator games to understand the circumstances under which respondents are willing to “put their money where their mouth is” to reveal their commitment to a sentiment (Forsythe et al. Reference Forsythe, Horowitz, Savin and Sefton1994; Engel Reference Engel2011). Our overarching approach to estimating the effect on attitudes through multiple measures thus allows us to understand not only the extent to which variation in media tone affects attitudes, but also which dimensions of attitudes it affects.
While our principal interest is in gauging the causal connections between the tone of articles and attitudes toward social groups, we also seek to understand the underlying mechanisms linking article tone to attitudes. We recognize that there are a wide range of potential moderators and mediators that may be relevant, and we do not attempt to test all of them. Our goal, instead, is to examine the potential role of anxiety as a key mediator of any valence effect. We focus on anxiety because scholars have shown that negative news is linked with negative affect (Johnston and Davey Reference Johnston and Davey1997; de Hoog and Verboon Reference Hoog and Verboon2020), and that group cues about potentially threatening information increase anxiety (Brader et al. Reference Brader, Valentino and Suhay2008). Once activated, anxiety reinforces a tendency to seek out, remember, and agree with negative information (Gadarian and Albertson Reference Gadarian and Albertson2014). Anxiety thus offers a theoretically and intuitively plausible mechanism linking the valence of newspaper articles and attitudinal responses, the effects of which may or may not vary depending upon the group associated with coverage. If articles with a negative valence increase anxiety in our participants compared to positive valence, then we hypothesize that this induced anxiety will make attitudes less positive.
Methods and Data
To estimate the effect of media tone on attitudes, we draw on a dataset produced by Bleich and van der Veen (Reference Bleich and van der Veen2021) that uses lexical sentiment analysis to measure the tone of articles published in 17 national and regional US newspapers between 1996 and 2016 that contain root words related to Muslims and Catholics. This dataset calibrates the results for each article against a corpus of articles that constitutes a representative sample of the US print media, the mean and standard deviation of which are set to 0 and 1, respectively.
The average valence of articles mentioning Muslims is markedly lower than that of articles touching on the other groups, in that they are both statistically and substantively more negative. The average article mentioning Muslims has a tone of −0.83, whereas the average for Catholic articles is 0.18. Readers of US newspapers are thus exposed to much more negativity and to much less positivity when reading about Muslims than when reading about Catholics. The extent of the negativity is large in comparison not only to Catholics, but also when compared to articles about Latinos or Hispanics, where the average article tone is 0.02 (Bleich et al. Reference Bleich, Callison, Edwards, Fichman, Hoynes, Jabari and van der Veen2021). The substantial negativity is not, however, unique to the American context, as corpora of Muslim articles drawn from the British, Canadian, and Australian press exhibit a similar average tone (Bleich et al. Reference Bleich, Souffrant, Stabler and van der Veen2018).
Figure 1 displays the overall distributions of valences for articles about Muslims and Catholics in the Bleich and van der Veen (Reference Bleich and van der Veen2021) corpus compared to that of the representative corpus. Only 1% of the 236,066 articles in the Muslim dataset have a valence of + 2 or higher, making these extremely positive articles rare, especially compared to the 14% of articles that have a valence of −2 or lower. A Muslim article with a tone of 0 is more positive than 78% of all articles mentioning Muslims or Islam in the dataset. This means that although such an article is neutral in comparison to a representative sample of American newspaper articles, it is relatively positive compared to the average article about Muslims. Among the 352,627 articles that mention Catholics, the average article is modestly positive. Only 3% of all Catholic articles have a valence greater than + 2 and only 2% have a valence more negative than −2.
Figure 1. Valence distribution estimates (kdensity) by corpus.
To explore whether articles of different valences affect attitudes, we conducted an online between-subjects experiment exposing readers to real published articles about Muslims or Catholics at one of three separate valence levels. We selected two articles each with a tone of −2, 0, and + 2 and an average word count for all articles in the Bleich and van der Veen (Reference Bleich and van der Veen2021) Muslim corpus, or approximately 550 words.Footnote 7 Articles with a tone of −2 or + 2 have a clear negative or positive valence, respectively, whereas those with a tone of 0 have no clear valence relative to the average US newspaper article.
Our treatment variation thus comes from differences in the valence of actual newspaper articles. We considered using baseline articles at each valence level and simply changing the name of the relevant group. This proved impossible, given that news articles are typically set in a context that is specific to that group. We could not, for example, substitute “Muslim” for “Catholic” in an article about church pedophilia scandals, or “Catholic” for “Muslim” in an article about a religious festival in Iran. We acknowledge that the non-comparability of topics across treatments is a drawback, but for our purposes the advantages of enhancing external validity and representativeness by using actual newspaper articles outweigh the limitations of doing so for testing the effect of article valence on attitudes (cf. Brunswik Reference Brunswik1956; Druckman and Kam Reference Druckman, Kam, Druckman, Green, Kuklinski and Lupia2011).
Articles either mentioned Muslims or Catholics, but not both, nor any other religious group, and each participant was exposed to just one of the six treatments. While this required us to gather a large sample, it obviated a concern that exposure to multiple treatments could artificially inflate the effects we observe via experimenter demand (Zizzo Reference Zizzo2010). The instructions employed and the articles used are provided in the online Appendix.
To obtain a large sample from a relatively diverse population within the United States, we implemented the experiment as a Qualtrics survey on Amazon's Mechanical Turk platform during February and early March, 2019. While Mechanical Turk respondents are not representative of a national sample, our experiment focuses on the effect of media exposure on individual attitudes and thus does not require that our respondents perfectly reflect American demographics. Participants were told the task would last approximately 15 minutes and for completing the task they would be paid a fixed payment of $1.5 but could earn a bonus of up to an additional $2. In total, we gathered 1013 complete observations, the participants who finished the experiment spent an average of 9.32 minutes on it, and they earned an average of $2.65, including the bonus.
To assess participant attitudes toward members of religion when the stakes are low, we deployed a “feeling thermometer” survey question that asked, How do you feel about (Muslims/Catholics)? Respondents could answer with scores between 0 (very cold or unfavorable) to 100 (very warm or favorable). This is our primary measure of attitudes toward each group.
To test the robustness of the media treatment effect on attitudes, we raised the stakes by including a financial incentive. In this variant of the dictator game, participants are given an endowment of $1, which they can keep, and are asked to donate as much as they want to a religiously affiliated charity. We drew from the websites of Islamic Relief USA and Catholic Charities USA to provide equivalent information about their charitable aims. In this case, if participants have a favorable view of the religion, they may elect to donate a portion of their bonus money to demonstrate this sentiment.Footnote 8
To deal with any order effects, we blocked the experimental outcomes.Footnote 9 In one block, comprising 611 of the 1013 complete observations, participants first faced the feeling thermometer, then the donation task. In the second block (used to gather the remaining 402 observations), participants first performed the donation task, then the feeling thermometer. For the analysis reported in the next section we pool the data and control for the order of the tasks.
After collecting our sentiment measures, we collected demographic controls, including the sex, age, education, race, and ethnicity of our participants. To account for other factors that may affect our analysis, we also asked participants about their primary source of news (i.e., from newspapers, TV, or radio) and their political leaning (on a scale from very liberal to very conservative). Lastly, to control for the overall optimistic or pessimistic outlook of participants and its potential effect on attitudes, we asked them to make an incentivized prediction (for which they could earn another $0.5) concerning something we posited would be orthogonal to the design of our experiment: the price path of an unnamed stock over 30 consecutive trading days. We picked the stock and time period so that a runs test would indicate that the price movements were essentially random, meaning there is no information to be inferred about the trading price of the stock on the 31st day based on the preceding 30 days. We therefore categorize people as generally optimistic/pessimistic if they predicted that the price rose/fell on the 31st day.
To measure anxiety, we gathered responses to the six-question State-Trait Anxiety Inventory (STAI-6) developed by Marteau and Bekker (Reference Marteau and Bekker1992). This instrument asks participants to respond on a 4-point Likert scale (from not at all to very much) to six statements about how the respondent feels right now. The statements include items such as, I feel calm, I feel upset, and I feel worried. We pooled the responses we collected from this instrument and used factor analysis to score our respondents' post-treatment anxiety levels. The factor loadings suggested that responses to the statement I am tense contributed most to the variation in the scores. For the analysis, we created an indicator variable identifying those participants with positive factor scores as “Anxious” (compared to those with negative scores, who are relatively unstressed) to facilitate the interpretation and graphical representation of our results.
The IRB-approved experimental protocol involved participants first giving consent to participate in a study about “reading comprehension.” They were randomized into one of the six treatments and asked to read two news articles. They were then asked a simple reading comprehension question. Only those who correctly answered this question were allowed to proceed. The next stage of the experiment involved the attitudinal tasks. Once sentiments were recorded, participants did the neutral belief elicitation about the stock price time path and answered the remaining survey questions. Within 3 days of the experiment, all participants received any bonuses they were due.
Regarding the statistical power of our study, we collected 1013 complete observations in six treatments for an average of 169 observations per treatment. Using a standard one-way ANOVA test, we calculated that at the 5% significance level and 80% power, our design should be able to identify a minimum effect size of a tenth of a standard deviation (0.11 to be precise). By all accounts, our sample should be sufficient to identify even small differences in the effects of our treatments.
Results
A summary of our experimental implementation and a description of our participants can be found in Table 1.Footnote 10 Overall, 40% of our participants were female, the average age of the participants was 35.38 years, 15% of participants had no more than high school education, 10% had an advanced degree (professional or doctoral), 10% were Hispanic and 80% were Caucasian. About half of our participants get their information primarily from newspapers, another 30% get it from television, 5% get it from radio and 17% get it from another source. Considering political affiliation, 40% of our participants self-identified as liberal and 20% as conservative to one degree or another.Footnote 11 In addition, 45% of our participants believed that the stock price rose in the neutral belief question they were asked.
Table 1. Average participant characteristics and treatment balance
Note: Of the 225 between treatment comparisons, 11 (or 5%) are significant at the 5% level.
To assess treatment balance, we conducted a series of t-tests to determine whether the average characteristics of our participants varied significantly between treatments. As the note in Table 1 indicates, of the 225 pairwise comparisons conducted, only 11 were significant. In fact, this is exactly what we would expect from randomization—about 5% of the differences are significant at the 5% level.
Pooling across religions and valences, the mean attitude was 53.64 (S.D. 29.49) out of a possible 100. Figure 2 illustrates the attitude by treatment. The first thing to note is that, on average, attitudes toward Muslims are higher than attitudes toward Catholics. This result is confirmed in Table 2 in which we see the point estimate in column (1) is very close to 10 “degrees” (9.93/53.64 = 19% of the overall mean), the difference is always highly significant (p < 0.01) and this effect is robust to adding all the controls listed in Table 1. Our initial exploration suggests this is primarily a function of the far greater positivity toward Muslims of the 40% of our respondents who self-reported as left-leaning compared to the 20% of right-leaning respondents in our survey. As noted above, for the purposes of the current study we are interested in treatment effects within each religious group, not differences between religious groups. Because we control for the political leaning of respondents within the Muslim treatments and within the Catholic treatments, we are confident that this result does not affect our findings.
Figure 2. Does valence affect attitudes?
Table 2. The effect of media valence on attitudes
Notes: OLS; dependent variable is feeling thermometer attitude; robust standard errors; * p < 0.10, ** p < 0.05, *** p < 0.01.
Concerning the primary hypothesis of our study, Figure 2 indicates that valence does affect sentiment. The Catholic treatment makes this relationship particularly clear. Reducing the valence of the newspaper articles from the neutral 0 baseline valence to −2, reduces the attitude of participants by 6.39, or 13%, on average. Somewhat symmetrically, increasing the valence from 0 to +2, increases attitudes by 6.98 or 14%. For the Muslim treatments, the reduction from neutral 0 to −2 also results in a decrease in attitude by 8.54 (14%) but a valence increase of the same magnitude has no noticeable effect on attitudes: the difference is a small decline of −0.74 or 1%. Hence, article valence seems to matter, but in ways that differ by religion.Footnote 12 The limited effect of positive Muslim articles during our 2019 experiment is consistent with Lajevardi's (Reference Lajevardi2021) findings from her 2016 treatments, though not with the greater impact found in her 2018 experiment or with the modest results from counterstereotypic treatments by Saleem et al. (Reference Saleem, Prot, Anderson and Lemieux2017) and Williamson (Reference Williamson2020).
To confirm the main valence results shown in Figure 2, consider the two regressions reported in Table 2 in which we account for valence continuously.Footnote 13 In the first column we report the raw differences without any controls, and estimate that a unit increase in valence increases attitudes in the Catholic treatments by 3.34 units (p < 0.01). The gradient is 1.40 (p = 0.21) shallower for the Muslim treatments, but as displayed in the Val + (Val × Mus) row, the resulting Muslim slope of 1.94 is still significantly positive at the 5% level. After controlling for the observables we collected, column (2) of Table 2 shows that the estimate of the Catholic slope increases slightly (to 3.79). The gradient of the Muslim treatments is 1.93 units shallower and this difference is now significant at the 10% level; however, these two effects balance each other so that the Muslim valence slope is about the same as in column (1) at 1.87 (p = 0.02).
In sum, the gradient is steeper for respondents' views of Catholics than it is for views of Muslims, but in both cases, being treated with articles at different valence levels has a direct effect on attitudes about these two groups. In categorical terms, readers exposed to articles with a negative valence report less warmth compared to respondents who read neutral articles, a result that applies equally to Catholics and Muslims. Yet, there are also differences across groups. Compared to readers of neutral articles, those exposed to articles with a positive valence report more warmth toward Catholics, though the same differential is not found among readers of Muslim articles.
Robustness: An Alternative Specification of Attitudes
If the effect of media tone is clearly visible on participant attitudes, what happens when the financial stakes are raised? Turning to the second outcome measured in the experiment, 29% of our participants donated to one of the two religious charities. The donations varied from $0 to $1 and averaged $0.12 (S.D. 0.25). Interestingly, the donations are larger to the Muslim charity, on average. This difference, estimated in Table 3, is considerable. Here we see that people in the Muslim treatments donate approximately 7 cents more (p < 0.01), which, compared to the combined mean of 12 cents, is a 58% increase.
Table 3. The effect of media valence on donations
Notes: OLS; dependent variable is charitable donation; robust standard errors; * p < 0.10, ** p < 0.05, *** p < 0.01.
Despite people having to pay to express their sentiments in the donation task, there is still an effect of article valence. In Figure 3, we see valence effects that are similar in shape to the attitudes depicted in Figure 2 with some large relative differences. Starting with the Catholic donations, moving from the 0-valence baseline to a valence of −2 reduces donations by 4 cents, on average, which is a 44% decline; increasing valence from 0 to +2 results in a 3-cent average increase, or 32%. The results are more modest for the Muslim treatments, however. Lowering valence from 0 to −2, in this context, reduces donations by only a fraction of a cent (−0.002); increasing valence from 0 to +2 results in a larger increment of 3 cents, or 23%.
Figure 3. Does valence affect donations?
In Table 3 we see that the valence gradient for the Catholic donations is, indeed, positive and the slope is significant at the 1% level. In column (1), without controls, we find that a unit increase in valence results in almost a 2-cent increase in donations; when we add controls to the regression in column (2) this estimate increases slightly. By contrast, the valence gradient is less steep for the Muslim treatments. Our estimate of the gradient in the Muslim treatments itself is reflected in the sum of the Val and (Val × Mus) lines (i.e., 0.009). It is invariant to the addition of controls and is not significantly different from zero. The bottom line is that donations depend on media valence for Catholics but not for Muslims. This offers some support to the view that media tone can affect attitudes even when money is at stake, but suggests that the extent of the effect may depend on the group in question. As with our findings with respect to feeling thermometer attitudes, the effect is clearer and more internally consistent for Catholics than for Muslims. Compared to the neutral baseline, positive Muslim articles had more of an impact on donations than their negative counterparts, though neither was statistically significant. These differences further emphasize the complex and cross-cutting nature of counterstereotypic information on attitudes and actions toward religious groups (Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Williamson Reference Williamson2020; Lajevardi Reference Lajevardi2021).
Anxiety as a Potential Mechanism
The last aspect of the experimental data to consider is whether anxiety mediates the causal effect of valence on participant sentiment. Recall that the basic hypothesis is that negative media valence causes anxiety and anxious participants have lower sentiments about the reference religion. In contrast, positive valence is hypothesized to have the opposite effect. To explore this mechanism, we offer two tests, both based on Sobel (Reference Sobel1982). In the first test, we examine mediation piece-wise. First, we test whether valence affects anxiety among our participants and then we see how the link between valence and outcomes is attenuated once we control for anxiety.
Figure 4 indicates that there is, indeed, a strong relationship between valence and the reports of anxiety coming from our participants. Our baseline expectation is that because of randomization, Figure 4 should be essentially flat; instead, we see that participants report being more anxious when the valence is −2 than when it is 0 and more anxious when the valence is 0 than when it is +2. That is, our valence treatments appear to be inducing or reducing anxiety as hypothesized. More specifically, in Table 4 we estimate that the incidence of being “anxious” falls by 5% points for every unit increase in valence and this effect is highly significant (p < 0.01). Further, though the coefficient on the interaction term indicates that the slope is not quite as steep for the Muslim treatments, that estimate is not statistically significant. Lastly, controlling for the observables we gathered in column (2) has little effect on these estimates.
Figure 4. Does valence affect anxiety?
Table 4. The effect of media valence on anxiety
Notes: OLS; dependent variable is anxiety indicator; robust standard errors; * p < 0.10, ** p < 0.05, *** p < 0.01.
In Table 5 we assess what happens to the relationship between valence and sentiments when we control for anxiety (and all the relevant interactions). In these estimates, we see three important results that indicate mediation. First, when we add anxiety to the regression it is highly significant—anxious individuals have attitudes that are 10.07 degrees cooler than those who are not anxious (p < 0.01). Second, when we compare the coefficients on valence in columns (1) and (2), we see that the second is considerably lower, suggesting that part of the variation originally soaked up by valence in the Catholic treatments is actually due to the effect of valence on anxiety. The fact that almost a quarter of this valence effect is accounted for when controlling for anxiety is strong evidence that increasing or reducing anxiety is an underlying mechanism. Third, we can also compare the magnitudes of the estimated Muslim gradient in the two regressions. Without Anxious in the regression (column 1) we find that the valence gradient for readers of articles about Muslims—Val + (Val × Mus)—is significantly positive, but the inclusion of Anxious (in column 2) reduces this gradient to almost zero. Hence, we find evidence that induced anxiety drives the effect of article valence to a large extent in both religion treatments.Footnote 14
Table 5. Does anxiety mediate the treatment effect (Is induced anxiety a mechanism)?
Notes: OLS; dependent variable is attitude; robust standard errors; * p < 0.10, ** p < 0.05, *** p < 0.01.
In addition to the results in Table 5, direct testing for mediation using the Sobel–Goodman test leads us to similar conclusions. Here the estimates of the proportion of the total valence effect that is mediated by anxiety are 0.21 for attitudes and 0.16 for donations. We get similar results when conducting an analysis that uses the raw anxiety factor scores instead of an anxiety indicator for those with positive factor scores.Footnote 15 In short, anxiety significantly mediates the effect of valence on attitudes by every measure.
Discussion and Conclusions
This article offers four core contributions to the existing literature on the effect of media representations on attitudes toward Muslims. First, we show that varying article tone affects the attitudes of our participants. Second, we demonstrate that this pattern holds not only for Muslims, but also for Catholics, a “most different” religious minority group in the US context. Every unit increase in valence generates a predicted increase of approximately 1.9 degrees on the feeling thermometer scale for Muslims and roughly 3.8 degrees for Catholics. Third, our results show that incentivizing tasks by offering personally costly opportunities to donate to charities does not eliminate the effect of valence. Finally, we demonstrate that the mediator of anxiety plays a significant role linking the valence of articles to attitudes.
There are, however, important nuances to our findings. For instance, while valence matters for articles about groups as different as Muslims and Catholics, the precise effect differs by religion; it matters in a more linear fashion for participants who are primed with Catholic news compared to those exposed to Muslim news, whose reactions are kinked. In addition, our incentivized task revealed similar valence effects, but ones that were statistically significant only for respondents to the Catholic treatment. All in all, as we expected, the effects of media valence on attitudes and actions are more straightforward for a mainstream group and more complex for Muslims as a racialized outgroup for which individual attitudes may be more firmly entrenched.
Our research approach has certain unique advantages, yet it also has limitations. For example, our choice to use actual rather than constructed articles about each group enhances the plausibility of treatments but comes at a cost to internal validity. In addition, while we find a similar relationship between article valence and attitudes based on treatments about groups as different as Muslims and Catholics, our findings are not identical given that patterns are linear for Catholics and are kinked for Muslims. Future work can build on and complement our findings by drawing on a wider array of articles to detect more detailed patterns across a broader valence spectrum; it can investigate whether a study using constructed articles that maximize internal validity (correspondingly reducing the external validity we opted for here) produces similar results; and it can expand the number and type of groups beyond Muslims and Catholics to understand whether the effects we identify here apply to articles about other identity categories.
Our research also brings new evidence to bear on the question of whether positive portrayals of Muslims have as significant an effect on attitudes as negative representations. Studies based on television treatments have produced mixed results (Saleem et al. Reference Saleem, Prot, Anderson and Lemieux2017; Lajevardi Reference Lajevardi2020, 108–31, Reference Lajevardi2021). Our research raises the hypothesis that distinctive features of Muslim articles may help account for some elements of our findings. In the present study, equivalent numbers of respondents were exposed to neutral and positive articles as negative articles about our two groups. This frequency mirrors real-world coverage for Catholics, and is typical of many other groups in American society (Bleich and van der Veen Reference Bleich and van der Veen2021; Media Portrayals of Minorities Project 2020). It is, however, not typical for Muslims. As we note above, only 22% of articles about Muslims have a positive valence. An article about Muslims with a valence of 0 is thus neutral relative to all American newspaper articles, but it is far more positive than the average article about Muslims. This may explain the limited effect on attitudes when the valence of Muslim articles varies between 0 and +2, as both types of articles are positive when compared to the average Muslim article. At the same time, donations to Muslim charities rose only when respondents were exposed to articles with a valence of +2. It may take a rare and extremely positive set of articles about Muslims to elicit elevated financial contributions. These propositions require further, targeted investigation, but they suggest that the relationship between media tone and attitudes may be linked to the real-world distribution of valence in the media coverage of different social groups.
Muslims remain a highly stigmatized group in American and Western societies. Our study explores the causal effect of media valence on attitudes, revealing not only a clear link between them, but also a factor that accounts for a portion of the explanation. By investigating Catholic articles, we sought to understand whether the relationship between valence and attitudes also applies to a very different and more mainstream group that is associated with modestly positive coverage overall. Our results suggest that the media play a key role in constructing and reinforcing attitudes toward both Muslims and Catholics, and thus potentially toward a wide array of social groups within—and potentially beyond—the United States.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1755048321000328.
Acknowledgements
For feedback on developing the experimental protocol, we thank Martin Abel, Erick Gong, Peter Matthews, and David Munro.
Financial support
We gratefully acknowledge funding support from Middlebury College and William & Mary.
Replication data
Available from the authors upon request.
Erik Bleich is Charles A. Dana Professor of Political Science at Middlebury College. He is the co-author, with A. Maurits van der Veen, of Covering Muslims: American Newspapers in Comparative Perspective, Oxford University Press, 2022.
Jeffrey Carpenter is James B. Jermain Professor of Political Economy and International Law at Middlebury College. His research fields include Experimental and Behavioral Economics, and he is the co-author of Game Theory and Behavior, which is forthcoming at the MIT Press.
A. Maurits van der Veen is Associate Professor of Government at William & Mary. He is the co-author, with Erik Bleich, of Covering Muslims: American Newspapers in Comparative Perspective, Oxford University Press, 2022.
