INTRODUCTION
Clergy routinely use political speech in religious environments to provide political information, shape opinions on political matters, and mobilize their parishioners. While scholarship often focuses on the political and religious experiences and attitudes of individuals (Jelen Reference Jelen1993; Margolis Reference Margolis2018), the importance of churches and clergy remains central to the study of politics and religion (Wilcox and Larson Reference Wilcox and Larson2006). To date, a robust literature examines political activism in religious organizations primarily through surveys of religious leaders and congregants, case studies of particular churches, and qualitative work (Guth et al. Reference Guth1997; Djupe and Gilbert Reference Djupe and Gilbert2002; Deckman, Crawford, and Olson Reference Deckman, Crawford and Olson2008; Glazier Reference Glazier2015; Friesen and Djupe Reference Friesen and Djupe2017), showing that pastors engage in high levels of political communication in churches and that congregants receive and process those messages.
Despite the importance of the politics of pastors, we know much less about the actual political content of religious sermons. We provide a large-scale evaluation of the political messages that Protestant pastors provide to their congregants via sermons. Sermons provide clergy with an opportunity rarely given to other elites: to routinely engage in both topic and focus through spiritual instruction (Djupe and Gilbert Reference Djupe and Gilbert2003; Calfano, Oldmixon, and Gray Reference Calfano, Oldmixon and Gray2013). But how frequently do pastors use their sermons to talk about political issues? And which political issues do pastors feature in their sermons? To answer these questions, we examine the complete text of 110,423 sermons from 5,532 American religious (largely Protestant) leaders, using quantitative text analysis methods (Grimmer and Stewart Reference Grimmer and Stewart2013; Boussalis and Coan Reference Boussalis and Coan2016; Greene and O'Brien Reference Greene and O'Brien2016; Boussalis, Coan, and Holman Reference Boussalis, Coan and Holman2018; Muddiman, McGregor, and Stroud Reference Muddiman, McGregor and Stroud2019).
We find that sermons routinely contain political content: 37% of all the sermons in the dataset feature a political topic. Aggregate up to the pastor level, 71% of pastors in our dataset deliver at least one sermon with political content. Common topics include the economy, homosexuality, war, civil rights, welfare, and abortion. Our large-scale observational dataset builds on research that finds high levels of activism on core political issues through surveys of pastors and congregants (Deckman, Crawford, and Olson Reference Deckman, Crawford and Olson2008; Djupe and Friesen Reference Djupe and Friesen2018). We then validate our descriptive findings through crowdsourcing (Benoit et al. Reference Benoit2016), as well as semantic and predictive validation procedures (Quinn et al. Reference Quinn, Monroe, Colaresi, Crespin and Radev2010).
Next, we build on extant literature (Deckman, Crawford, and Olson Reference Deckman, Crawford and Olson2008; Djupe and Neiheisel Reference Djupe and Neiheisel2008; Calfano and Oldmixon Reference Calfano and Oldmixon2018) to examine which pastor, church, and location factors are associated with higher levels of political content generally and across specific political topics. We apply a novel approach to identify the community where our pastors preach by locating churches geographically and linking our sermons to census block and electoral precinct-level information. We draw on this demographic and political information, plus denomination and pastor race and gender to evaluate the relationship between community characteristics and pastors' political topics.
The results present a nuanced view of politics in religious environments. Pastors of most denominations, located in most communities, engage in some level of political discussion about most issues. Consistent with previous work, we find that Mainline Protestant pastors are more likely to discuss politics overall. Yet, contrary to expectations, Evangelical Protestant pastors are not more active on abortion, nor are they less active on social justice issues. Mainline pastors are more likely to talk about homosexuality (Thomas and Whitehead Reference Thomas and Whitehead2015). Overall, we find similar, albeit slightly lower levels of political speech as identified in most survey data (Deckman, Crawford, and Olson Reference Deckman, Crawford and Olson2008; Djupe and Neiheisel Reference Djupe and Neiheisel2008). Some of our findings emerge from the differences in data sources, including that our dataset provides an in-depth evaluation of Evangelical pastors' behavior.
Our study addresses a key methodological concern in studying political content in ordinary speech: how to address class imbalance, or when topics of interest appear only rarely in a text, which is a widely acknowledged and persistent problem for quantitative text classification (Japkowicz and Stephen Reference Japkowicz and Stephen2002). Here, we combine community-generated labels, crowdsourced codings, and a generative supervised learning model to overcome particular challenges associated with the detection of rare thematic content within large corpora, an approach that could be adapted by scholars studying a wide variety of types of text.Footnote 1 We also offer a more precise approach to evaluate whether the community surrounding a church shapes the political communication of the pastor. Measures of geography commonly employed by political science research cover large areas and are arbitrarily assigned (e.g., counties or cities); we go further, by building far more specific measures of the community surrounding each church, and introducing methodological approaches that could be broadly applied. Taken together, our results and approach provide a rich detail on the interwoven nature of religion and politics in the United States.
POLITICS IN THE CHURCH
Do clergy insert political content into their religious sermons? Which issues do they focus on? The extant research finds that most clergy report frequently presenting political content in sermons. In surveys, as many as 90% of pastors report putting issue-based political content into through their sermons (Guth et al. Reference Guth1997; Djupe and Gilbert Reference Djupe and Gilbert2002), although the share is lower when pastors are asked about whether they took a stand on political issues (56%) and whether that had happened in the previous year (30%). Generally, church-based issue activism focuses on moral issues and social justice concerns. Social justice concerns, rooted in the Social Gospel movement, include liberal frames of racial and gender equality, environmentalism, and welfare and social services, among others. Moral concerns, on the other hand, have their source in conservatism and focus on issues like abortion and homosexuality.
Surveys of clergy and the faithful and observational studies of sermons find clergy mention political content frequently but engage in position-taking (and use the pulpit for that position-taking) at far lower rates. For example, in survey data, 98% of clergy in Djupe and Gilbert (Reference Djupe and Gilbert2002) report addressing poverty and hunger and over 90% report addressing civil rights, environment, and education, with women's rights, unemployment and the economy, and gay rights also receiving significant attention. On moral issues, most pastors reported high levels of cue-giving relating to family issues, homosexuality, capital punishment, and abortion (Djupe and Gilbert Reference Djupe and Gilbert2002). Yet, these numbers reflect a wide range of engagement; Djupe and Gilbert count any and all pastor's responses, even if the pastor noted that they “only rarely” addressed the issue. Similar levels of political content were identified in observations of 95 different worship services (Brewer, Kersh, and Eric Petersen Reference Brewer, Kersh and Eric Petersen2003). In evaluating what the religiously faithful remember hearing, Scheitle and Cornell (Reference Scheitle and Cornell2015) find that 62% of respondents reported hearing about abortion and just under half heard content about homosexuality and the environment. However, the authors also note that most of these discussions are simply “passing mentions” in the service but not part of the sermon.
How might looking at the text of a large set of sermons line up or depart from the extant findings on pastor political activism? One possibility is that some (or all) pastors overreport their political activism in surveys because of social desirability biases (Presser and Stinson Reference Presser and Stinson1998), including that pastors believe that they should be talking about political issues more than they actually do. For example, pastors might feel pressure to report that they engage around the issue of poverty and the economy at higher rates than they do. Another possibility is that surveys overcount political content because of who is sampled in these surveys. For example, some of the samples used, including the surveys of ECLA and mainline Episcopal clergy (Djupe and Gilbert Reference Djupe and Gilbert2003), may be more liberal (and thus more politically active on some issues) than other denominations of clergy in the United States. Overcounting may also occur if pastors underreport the extent to which they engage in political cue-giving because they are concerned with how those reports will be received. Finally, our results might differ in that we can capture nuances in the political content of the speeches by not only looking at whether a topic is mentioned at all, but the share of the sermon that is dedicated to the topic.
VARIATION IN POLITICAL CONTENT ACROSS CHURCHES
While pastors often use political content, there is also variation in who is more likely to include these materials in their religious messages. Previous research has found that pastors respond to congregational and community preferences and denominational pressures and traditions, while also exerting their personal preferences in determining when and how to engage with political issues.
In general, clergy view their own role as primarily to tend to the flock by serving and preserving their congregation, including pleasing and serving existing members of her church and attracting new members (Djupe and Gilbert Reference Djupe and Gilbert2003). As such, clergy that view their congregations as politically active are more likely to be politically active themselves (Guth et al. Reference Guth2003). These findings suggest that the characteristics of congregations and the community have the potential to influence both the emergence and form of political content in sermons; as Djupe and Olson note, “While ministering to local concerns might simply involve a particular agenda construction, it also might entail identification with the particular values of the community first and the religious organization second” (Reference Djupe and Olson2010, 275; emphasis in original; see also Holman and Shockley Reference Holman and Shockley2017).
We expect that the socio-economic and political characteristics of the community around the church might shape political topics. Generally, individuals with more socio-economic resources, including income and education, are more politically engaged and active (Farris and Holman Reference Farris and Holman2014). These patterns would suggest that the income and education levels of the community will have a positive relationship with political content. Research also suggests that low socio-economic status areas attract clergy who are more interested in political change and social justice (Olson 2000). Clergy in areas with poor or minority residents may be particularly likely to discuss social justice issues. The political beliefs of the community may also shape pastor behavior, with more social welfare issues in liberal areas and increased moral concern topics in conservative areas.
Denominational affiliation is certainly important in determining the level of political content in sermons (Djupe and Gilbert Reference Djupe and Gilbert2003). Those denominations that focus more on the public role of religion and social gospel teachings, such as mainline Protestants, provide a natural frame for pastors engaging in some political issues in a religious context (Guth et al. Reference Guth1997). As such, mainline Protestant pastors may be more likely to feature political content, particularly those issues relating to social justice concerns. Evangelicals often focus religious attention on individual salvation and other-worldly concerns. On one hand, this would suggest that there will be less political content in Evangelical sermons, particularly when compared to mainline Protestant churches. On the other hand, Evangelical pastors may feel pressure to engage in politicking, particularly around moral issues. Thus, we expect that Evangelical pastors will engage more around moral issues, but less around social justice issues and general politics.
Clergy characteristics may also play a role in the emergence of political speech in sermons. In general, churches and religious leaders are well respected in the United States and are “important factors within American public life” (Smidt Reference Smidt2016, 2). Clergy also often have a significant degree of autonomy and can shape the foci of sermons within their congregations. Descriptive characteristics may also matter. Indeed, theories of descriptive representation suggest that characteristics like race and gender may shape the interests and foci of clergy. For example, female clergy are more likely to speak about gay marriage (Deckman, Crawford, and Olson Reference Deckman, Crawford and Olson2008; Friesen and Djupe Reference Friesen and Djupe2017) and Black clergy are more likely to engage in issues associated with race and civil rights. Women and minorities are attitudinally more likely to support social justice policies and are more likely to benefit from these policies (Holman Reference Holman2015; Ondercin Reference Ondercin2017).
Evaluating Political Speech in Sermons
To assess the degree to which political content in sermons map onto or depart from survey and observational data and which pastors might insert political content into their sermons, we face a methodological challenge: identifying this content. Past scholarship estimates the frequency of political speech in sermons by self-reports from clergy and members of congregations, or direct observation by researchers of religious environments (Guth et al. Reference Guth1997; Brewer, Kersh, and Eric Petersen Reference Brewer, Kersh and Eric Petersen2003). We supply an alternative and direct measure of religious speech: the content of sermons themselves. More specifically, we utilize a large set of sermons posted on SermonCentral.com (Woolfalk Reference Woolfalk2013), which is an online resource for Christian pastors to share sermons and religious leadership information with each other. SermonCentral itself claims to be the “largest” and “most popular” sermon research site in the world, with “over 250,000 church leaders” visiting the website every week.Footnote 2
On the website, pastors provide information about themselves, including their denomination, and contact information of their church (include the name and address), and a profile picture. Contributors label sermons with the key topics covered in the text (called “tags”), as well as the scripture referenced in the sermon. There is also a social aspect to SermonCentral, as other members of the community can add additional labels and scriptures, and comment on and rate the sermons. Like other social networks, we expect that SermonCentral is neither “democratic nor undemocratic,” in that it replicates existing inequalities in participation (Evans, Cordova, and Sipole Reference Evans, Cordova and Sipole2014; Barberá and Rivero Reference Barberá and Rivero2015). We do not know why pastors participate on the website, beyond what the pastors themselves say: they want an online community where they can share their sermons, communicate with other pastors, and learn from other religious elites.
DATA: A LARGE CORPUS OF RELIGIOUS TEXT
To examine patterns in the sermon data, we collected, cleaned, coded, and analyzed the posted sermons. In September and October 2015, we harvested the full text and meta-data for each sermon, resulting in a collection of 134,531 sermons from 6,716 unique contributors. We ignore sermons from churches located outside the United States or without a listed address, not in English, or from severely under-represented denominations (namely, Catholics and MormonsFootnote 3). We are left with 110,423 sermons from 5,532 pastors, covering the period October 2000 to September 2015. Figure 1 provides the geographic distribution of our churches. Within our sample, the average number of sermons uploaded per pastor is 19 (SD = 60).
Figure 1. Geographic distribution of churches in the sample. This figure displays the number of churches included in our sample by county
The sermons uploaded to SermonCentral are a convenience sample. We acknowledge that this sample is not representative, but we have reason to believe the pastors and their church locations look like U.S. Protestant churches and pastors. We benchmark our churches against the National Congregations Study (NCS), and the locations of our churches against the U.S. Census and Religious Census. Our sample provides a departure from previous studies as we can provide a more robust discussion of conservative denominations. Table 1 compares our sample to the U.S. population, a religious census, and on political variables.Footnote 4
Table 1. Benchmarking our sample against U.S. counties
The counties in our dataset resemble U.S. counties in terms of racial composition, income, poverty rate, education, and age, while our sample's counties have a higher average population than the average U.S. county. Our total rate of adherence is slightly lower than the average U.S. county, as is the mainline Protestant rate of adherence, while our sample counties have a higher rate of Evangelical adherence than the national average. We also compared the counties in our dataset that also appear in the 2012 Cooperative Congressional Elections Study (CCES) to counties in the CCES overall and find strong similarity in political party identification, the importance of religion, and church attendance, as well as on levels of support for Obama in 2008. These comparisons give us confidence that the churches in our dataset are located in demographically, religiously, and politically representative places.
To evaluate the ways that our sample resembles American Christian elites, we compare the location, denomination, and demographics of our pastors to the 2012–2013 NCS and to the 2010 Religious Census. We find that our pastors resemble those pastors in the NCS relatively well. Evangelicals make up just under 64% of our pastors. In comparison, Evangelicals make up 67.2% of pastors in the NCS. In turn, mainline Protestants are 22% of our pastors and 20.3% of the pastors in the NCS.
A COMPUTATIONAL APPROACH TO IDENTIFYING POLITICAL CONTENT IN SERMONS
The large size of our data provides an ideal opportunity to use scalable, computer-assisted methods to accurately and reliably classify political text (Grimmer and Stewart Reference Grimmer and Stewart2013; Aaldering and Van Der Pas Reference Aaldering and Van Der Pas2018; Boussalis, Coan, and Holman Reference Boussalis, Coan and Holman2018) including material from political elites (Grimmer Reference Grimmer2013; Greene and O'Brien Reference Greene and O'Brien2016). What is the best strategy for extracting political references from the corpus? One approach would be to construct a dictionary of theoretically-informed political keywords that would be used to calculate the prevalence of these keywords in our corpus of sermons. Yet, the nature of political content and religious texts presents technical difficulties for classifying politics using dictionary-based methods as many political keywords also have a religious meaning. For instance, consider the word “election.” While election is certainly present in discussions of electoral politics, it also appears quite regularly in the Christian concept of the “elect” or God's chosen people.
If simple dictionary-based methods are inadequate, what other tools are available to classify political speech in a large collection of sermons? The field of machine learning offers a wide range of statistical tools for meeting this objective. The most common approach to supervised learning methods (see Grimmer and Stewart Reference Grimmer and Stewart2013, 9) relies on discriminative models to predict categories or “classes.” When the underlying data takes on a simple structure, these models have been shown to be a superior tool to the other major classification approaches, namely generative models (Ng and Jordan Reference Ng and Jordan2002). Yet, discriminative models may perform poorly when some classes are much more prevalent than others—referred to as class imbalance—and there are a large number of potential classes from which to choose (Rubin et al. Reference Rubin, Chambers, Smyth and Steyvers2012). Unfortunately, these properties are defining features of our sermon corpus (and many other politically oriented texts). Political language is infrequent (and linguistically diverse) when compared to content relating to core religious themes. Even a cursory glance at the SermonCentral corpus reveals that the majority of a sermon communicates religious rather than political matters. Moreover, past scholarship demonstrates that American pastors engage with a large number of political topics when preaching to their congregants. For instance, Djupe and Gilbert (Reference Djupe and Gilbert2002) identify 16 broad topics in their survey of Lutheran and Episcopalian ministers. Even this small number pushes the limits of what is considered a reasonable level of performance for discriminative models (Liu et al. Reference Liu2005; Rubin et al. Reference Rubin, Chambers, Smyth and Steyvers2012).
Our approach addresses these challenges by combining the pastor-generated tags with human coding and a generative statistical model for text data. The user-generated labels capture the key topics or themes associated with each sermon. A sermon entitled “Bringing America Back to God” was tagged by users of SermonCentral with the following labels: economy, America, revival, socialism, morality, and the scripture Judges 1:1–21:25. Using the labels as a starting point for classification is easy, given that they capture the primary political themes described in past literature. Nevertheless, labels must be standardized prior to analysis and we had to determine which labels are actually political. Given the free-form nature of the labels and the size of the sermons corpus, we harvested the list of tags (N = 19,525) and employed crowd-sourcing to reduce the label set from 19,525 to 231 tags that could be about politics (Benoit et al. Reference Benoit2016), a process described in the Appendix. Next, we assess each of the 231 labels by reading the content of five sermons associated with each label. Finally, we aggregate labels into 25 themes that include common political topics, such as combining the tags “patriotism” and “4th of July.”Footnote 5
Learning About Political Speech From Community-Generated Labels
After we identify the relevant political labels, we face a challenge: we cannot directly use the standardized labels to measure political communication by, for example, counting the number of times a particular label appears in the corpus. The observed labels often fail to be attached to politically relevant sermons. For instance, a sermon entitled “What about homosexuality?” is clearly about homosexuality, yet, the only generic tag on the sermon is “sin bondage to.” Put simply, while sermons tagged with political labels almost always contain politically relevant content, the converse is not necessarily true. Just because a sermon does not have a political tag does not mean it does not contain political content.
To apply these missing labels, we rely on a generative model to infer missing labels from observed labels, employing a supervised extension of the well-known latent Dirichlet allocation (LDA) outlined in Rubin et al. (Reference Rubin, Chambers, Smyth and Steyvers2012)). The labeled or “flat” LDA provides a simple, hierarchical Bayesian model for the random process responsible for “generating” a particular text. After performing a series of text pre-processing steps,Footnote 6 we estimate the labeled LDA, first training the model using k = 500 iterations. Next, we treat all the sermon labels as missing and infer their topics based on the parameter estimates from the trained model. The topic model provides to us a set of topics, which we then label based on the most probable keywords for each topic and by reading sermons with large estimated proportions of a given topic. Moreover, the model also estimates the proportion of words devoted to each political theme in each sermon. These proportions may be aggregated to calculate the relative prevalence of each political topic in the overall corpus. For technical details on the L-LDA model, see Appendix B.
THE POLITICAL TOPICS IN SERMONS
How often do political topics occur in sermons and which political topics occur most often? Figure 2 provides an illustration of the politically relevant topics that were classified by the model within the sermon corpus. Specifically, the figure displays two interpretations of topic prevalence. The left panel illustrates the proportion of sermons where a particular political topic is among the three most probable topics in a given sermon. The most prevalent of the 21 political topics, according to this metric, is Economy which is among the top three most probable topics, appearing in 7.2% of the sermons in our corpus. We also find that Welfare, War, Homosexuality, Evolution, Abortion, and Civil Rights are among the most prevalent of the political topics.
Figure 2. Political topic prevalence. The left panel shows the proportion of sermons where at least one political theme is among the three most probable topics for that sermon. The right panel displays the proportion of pastors who delivered at least one sermon where a given political topic was among the top three likely topics. Topics in the left panel are sorted in descending order of prevalence, while the right panel follows the same variable ordering
As an alternative metric, the right panel of Figure 2 displays the proportion of pastors who have delivered at least one sermon that contains a given political topic among the sermon's three most likely topics. We find very similar topic prevalence rankings when using this metric. We find that 32.3% of pastors authored at least one sermon that focused substantively on matters related to the Economy, while the next set of most prevalent political topics are essentially identical to those shown in the left panel. Using the pastor-level metric, we find that 70.7% of pastors delivered at least one sermon where any political topic was among the top three most likely themes. In some ways, these common topics map well onto survey research on pastors and the faithful. Past research has found high rates of reported content related to welfare and civil rights (Guth et al. Reference Guth1997; Djupe and Neiheisel Reference Djupe and Neiheisel2008), which are also common topics in our data. However, other topics where pastors report frequent engagement, such as the environment, education, and family problems, occur far less often in our results. We can also identify higher order groupings of these topics into meta-topics, such as moral issues, social justice, and general government content. Further, although we are working with a convenience sample of sermons, the fact that both traditionally conservative topics (e.g., homosexuality and abortion) and liberal themes (e.g., civil rights and welfare) emerge from our topic model suggests the presence of ideological heterogeneity among the sampled pastors.
What do these issues look like in the sermons? We read the sermons with the highest probability of any of our central topics to examine the framing of these issues. As one might expect, many sermons approach issues like Abortion and Homosexuality from a socially conservative perspective—with some more extreme in their outlook than others—but also with a clear political message to the congregants. For instance, one Baptist pastor concluded his sermon with the following piece of advice:
We must be politically involved and vote pro-life, no matter if you are a Democrat or a Republican. How dare we cast votes for people because of economic reasons, social security, or any other reason and not stand up for the unborn!
Homosexuality is often framed as a sinful act and not as an identity. For example, an Evangelical pastor described those who are “caught in the sin of homosexuality” as being gay by choice and that they can change their “lifestyle” at any given moment. From a political point of view, homosexuality is framed as a product of public policy encroaching on traditional family values. Same-sex marriage is largely condemned. In general, sermons that are classified as highly probable of containing these moral issue topics often express a sense of moral warfare in American politics and society. As another sermon notes:
As we move into the 21st century, should the Lord tarry, I think probably the next civil war fought in America, not fought with guns and bullets, but fought with editorials and laws, will be the fight over abortion and acceptance of homosexuality.
Social justice themes are also present in the corpus. Sermons with a large share of words that are related to Civil Rights emphasize the virtue of racial equality. For instance, after clarifying that racism is still a scourge on American society, one pastor provides a religiously inspired condemnation of racism:
One of the greatest weapons of Satan is division. Simply because if he can divide he conquer, and the fact is, there is only one race, the human race! People may have white skin, black skin, brown skin, yellow skin, red skin, or any other color skin, they may have different ethnic backgrounds, but they are all part of the same race, the human race!
Welfare and poverty content often focus on explaining how God could allow people to be poor and how to learn from observing poverty. Other pastors talk about how wealth and poverty are not signs of Godliness. Use of the Good Samaritan is common, as are religious obligations to care for the poor. For instance:
The truth of the matter is it's not enough to give charity. It is our duty to do our share to see to it that we build a society where charity will not be unnecessary, a society where no sick person will go unattended, no hungry person will go unfed, no one is poorly housed, no able-bodied person will go without adequate employment, and good schools will be provided for all.
Other political content ranges widely from connections between war and biblical stories to the encouragement of parishioners to turn out to vote in order to condemn specific acts by the government or political leaders. Overall, these results suggest a wide range of political content in religious speech (and ideological diversity in that speech), but that most pastors frequently give sermons without any political content.
MODEL VALIDATION AND EVALUATION
How else might we demonstrate that these topics are actually capturing political content in sermons? The descriptive labels and manual review of sermons with a high probability of containing a given political topic that is offered above provide an initial demonstration of the semantic validity of the model.Footnote 7 In this section, we offer further support for this validity criterion by: (1) examining the degree to which the estimated political topics meaningfully relate to one another (i.e., semantic similarity) (Quinn et al. 2010); and (2) whether the attention given to certain political themes over time corresponds to relevant external events (i.e., predictive validity).
Semantic Similarity
Given that a “topic” in the L-LDA model is a probability distribution of words given the topic, we can represent the “semantic similarity” between topics as the distance between probability distributions. In this study, we accomplish this task by measuring the Jensen–Shannon Divergence (JSD) between topics (see Endres and Schindelin Reference Endres and Schindelin2003). Similar topics share a smaller JSD value, while dissimilar topics have larger JSD values. The top panel of Figure 3 illustrates the semantic similarity of the estimated topics by mapping the pairwise JSD distances onto a two-dimensional space via multidimensional scaling. The 21 politically-relevant topics are labeled and represented by blue circles, while the remaining 40 religious topics are shown as gray squares.
Figure 3. Topic labels and similarities. The top panel of this figure illustrates topic similarity by displaying Jensen–Shannon distances which are projected onto a 2D space with the use of multi-dimensional scaling. Labels marked as blue circles correspond to political topics, while gray squares relate to religious themes. Topics that use similar words are closer together and vice-versa. The bottom panel displays the full list of estimated political topics (in alphabetical order) and provides the top 5 stemmed keywords for each topic
The relationships between topics conform with the discussion of meta-themes found in the literature. First, we can discern a clear cluster of most religious topics (the “Religious core”) which is semantically distinct from the political topics, meaning that the model picks up semantic differences when pastors engage with entirely religious versus religious-political themes. Second, among the political topics, interesting clusters emerge. Along the bottom portion of the figure lie topics related to American government and security (e.g., Founding, Liberty, Patriotism, Elections, American values, Terrorism). Surrounding the religious core are political topics that are typically related to moral issues (e.g., Abortion, Homosexuality) and social justice (e.g., Civil rights, Welfare). Moving along the top portion of the figure, we can see political topics that involve more general issues (e.g., Crime, Education) and science (e.g., Environment, Stem cell). Overall, we are satisfied with how such a simple model, based on word co-occurrences, can distinguish political speech from religious preaching and allows for the emergence of meaningful higher-order clusters of political themes.
Predictive Validity
We next validate the data by comparing the prevalence of each topic to external events, essentially evaluating the predictive probability of the topic models (see Quinn et al. 2010). We should expect elevated content of a given theme during a period of time when a particularly relevant event has occurred which would prompt cue-giving relating to the topic. Figure 4 displays the mean topic probability of four political themes (General politics, Economy, Abortion, and Homosexuality) for each yearly quarter over the period 2000q4 to 2015q3, with quarters darkened where we would expect higher values, given real-world events.
Figure 4. Predictive validity of selected topics. This figure illustrates the average quarterly topic proportions of four political topics (General politics, economy, abortion, and homosexuality) over the period 2000q4–2015q3. Periods of interest are highlighted in red
How well do our topics perform in terms of predictive validity? Beginning with the Politics (General) topic, which is a summation of all 21 political topics estimated by the model, we see that spikes in content seem to coincide with Presidential election seasons (2004, 2008, 2012) as well as salient national events (e.g., 9/11). How about the Economy? We find a bulge in the discussion of economic-related themes around the time of the 2008 financial crisis. For Abortion, we see an escalation of speech around major court cases: a series of federal cases in Q1 and Q2 of 2004 (including Nat'l Abortion Fed'n v. Gonzales, Ayotte v. Planned Parenthood in Q1 and Q2 of 2006, and Gonzales v. Carhart in Q2 of 2007). Further, we see increased content relating to Homosexuality around the time of the first legal gay marriage in the United States in May 2004 and the Obergefell v. Hodges ruling in June 2015, which essentially legalized gay marriage across the United States.
Overall, relying on semantic and predictive validation procedures, we are confident that our topics are picking up the underlying subject matter within the corpus in a consistent and cohesive manner. At the same time, there is noise in these figures, with increases and decreases in content outside national events, suggesting that pastors employ political cues not just when issue salience increases, but also because of their preferences and local concerns.
WHAT FACTORS ARE ASSOCIATED WITH DIFFERENCES IN PASTORAL SPEECH ON MAJOR POLITICAL THEMES?
Given the high level of variation that we find in the content of overall political issues and on individual topics, what pastor-, church-, and community-level factors help explain if and how pastors discuss politics? We examine this question by combining salient political themes and meta-data from the SermonCentral corpus with demographic and electoral data for each church's surrounding area.Footnote 8
Dependent Variables: A Set of Politically Relevant Themes in Sermons
We examine the overall level of political content (Politics (General)), as well as five sub-topics: the Economy, two moral concern topics (Homosexuality and Abortion), and two social justice topics (Civil Rights and Welfare) as our dependent variables. In each case, we measure the level of content as the proportion of words in each sermon that the L-LDA model assigns to a relevant topic. Further, we generate a general politics variable (Politics (General)) which is measured as the proportion of words in a sermon that are devoted to any of the politically-relevant topics in Figure 3 (see blue circles).
Pastor, Church, and Community-Level Data
Our primary independent variables include information about pastor, church, and the area surrounding the churches. First, when considering church- and pastor-level factors, we utilize a variety of self-reported information from the clergy, including denominational information from each pastor, which we aggregated into general categories according to the Handbook of Denominations in the United States. We coded race and gender information using photos and the U.S. Census' list, which provides gender and race probabilities of each first and last name of those names where there are more than 100,000 people with that name in the United States (see Sumner Reference Sumner2018).Footnote 9
Second, we collected information on key demographic and economic information for areas surrounding our sample of churches from the U.S. Census. However, defining the appropriate area or “neighborhood” associated with a particular church using Census data for a geo-coded church is not straightforward. To provide a reasonable estimate of the areas that could supply potential congregants to a given church, we relied on measures from the U.S. Department of Transportation 2009 Household Travel Survey (Santos et al. Reference Santos, McGuckin, Nakamoto, Gray and Liss2011, 13), which found that the average person trip length to “school/church” is 6.3 miles. Using this information, we defined a church's “neighborhood” as all of the Census block groups intersecting a 10 mile radius around the church (for more information on the construction of the geographic measures, see Section D of the Appendix). With the neighborhood estimate in hand, we aggregated—using the mean—information on the following variables: natural log of the total population (Geo: Pop. (ln)), proportion of white residents (Geo: White), and median income (Geo: Income). To measure the political leaning of a given neighborhood, we also incorporate voting data from the 2008 Presidential election (Ansolabehere, Palmer, and Lee Reference Ansolabehere, Palmer and Lee2015) and include all precincts that intersect the 10 mile radius for the church. We use the percentage of the two-party vote for President Obama in the 2008 election (Geo: Obama 08 %). Lastly, we record whether a given church is located in the U.S. South (Geo: South).
Statistical Methods
We utilize a fractional logit model because of the number of features of the SermonCentral data that complicate model selection. Our dependent variables are based on the proportion of words devoted to a particular topic and necessarily range from 0 to 1, which precludes the use of OLS regression. In addition, we use sermon-level data with nested observations within pastors and communities. We operationalize the church “community” as the union of churches with overlapping neighborhoods (see Section D of the Appendix for more information on constructing church communities). To address these two unique features of our data, we employ the fractional logit model, which is well-suited for dependent variables bound between [0,1] (Papke and Wooldridge Reference Papke and Wooldridge1996). As such, the models presented below provide estimates from fractional logit models of sermon-level data, with clustering on both the pastor and church “neighborhood.”
RESULTS
What factors are associated with higher or lower levels of political content by clergy? Figure 5 (and a related table in Appendix G) illustrates the estimates from the series of fractional logistic regression models described above that seek to explain variation in a sermon's share of words that are related to politics generally as well as to a set of relevant political themes. The figure shows the coefficients (log odds) and confidence intervals (99 and 95%) of the covariates described in Section 6.2. The results are presented in thematic groups. Panel (a) displays the results of models that explain political content in general and relating to the economy. Panel (b) presents models which explain variation in sermon attention to moral issues such as abortion and homosexuality. Last, Panel (c) shows the results of models focused on the variation of speech on social justice matters such as civil rights and welfare.
Figure 5. Explaining variation of politically relevant discussion in sermons. This plot illustrates the results of the fractional logistic regression models. Standardized coefficients (log odds) and confidence intervals [99, 95, 90%] are displayed. Regular coefficients are displayed for dummy variables. The figure displays model results for six (6) dependent variables. Panel (a) shows the results for Politics (top, blue, square) and Economy (bottom, orange, diamond). Panel (b) displays the estimates for Abortion (top, red, triangle) and Homosexuality (bottom, light blue, circle). Panel (c) illustrates the estimates for Civil Rights (top, purple, diamond) and Welfare (bottom, green, square). Note that church size statistics are only reported in the Appendix
We first examine the models presented in Panel (a): general political and economy-related content within sermons. Starting with the results for General Politics, we do not find any pastor gender differences in the proclivity to politick in general. With respect to denomination relationships, there is no statistically significant difference in general political content between sermons from the Evangelical traditions with those of the “other Christian” baseline category (i.e., non-Evangelical and non-Mainline Protestant churches). However, Methodist and Presbyterian pastors employ a higher share of words that deal with political themes in their sermons. We find few effects for the location of a church except that sermons from churches in more political liberal locales contain more words related to politically relevant themes. In many ways, these findings depart from survey research, which often finds large denomination differences in who engages in political cue-giving.
The next model in Panel (a) estimates variation in the share of words related to the Economy. While there is little evidence of relationships between the gender or race of the pastor and sermon topics, similar to General Politics, we again find denomination variations. Namely, the results suggest that sermons from Mainline Protestant churches are positively and significantly related to the share of economic content, which is consistent with the extant scholarship. We also find no differences in the community; that is, pastors in poorer (or richer) places are no more likely to give sermons with economic content.
Next, we review the results of models in Panel (b) associated with the moral concern issues of Abortion and Homosexuality. Again, the results of the statistical tests suggest that denomination shapes patterns of these political topics. Specifically, mainline Protestant pastors from Methodist, Presbyterian, and Episcopalian sermons are positively associated with abortion content, departing from our expectations about Evangelical activism on the issue. We do find that churches from both the Evangelical and Mainline traditions are statistically different from pastors from other traditions in content relating to the issue of homosexuality. The population of the area around the church is negatively associated with discussions about abortion. Pastors in more affluent neighborhoods are more likely to mention abortion in their sermons. Further, the political liberalness of a church's locale has a positive relationship with the share of words within a sermon that are related to abortion and homosexuality. Sermons by Southern pastors have less content relating to homosexuality than their non-Southern counterparts.
The last set of models focus on the discussion of Civil Rights and Welfare in religious sermons, illustrated in Panel (c). We find little evidence of pastor demographic effects on social justice themes. However, the results suggest that white pastors are less likely (10% error level) to include welfare themes in their sermons. With respect to denomination relationships, Evangelicalism is associated with less content relating to civil rights and welfare, while Mainline Protestant denominations (particularly Methodists) are associated with a higher level of content about both civil rights and welfare than the baseline. There are mixed church- and locale-related results on the social justice issues content in sermons: churches in more liberal locales produce sermons with a greater use of words related to civil rights, while Southern sermons have more welfare content.
DISCUSSION
In general, churches and religious leaders are well respected in the United States and are “important actors within American public life” (Smidt Reference Smidt2016, 2). Understanding the political content within religious sermons is thus an essential component of understanding politics more generally in America. In this paper, we use a large corpus of religious sermons and a computational text analysis approach to examine these speeches. Our research provides a next logical extension to a robust body of scholarship that has evaluated pastor political behavior through surveys, observations, and interviews. We build on these avenues of research in our evaluations; in turn, our results could inform future survey, observation, and interview research.
Methodologically, we provide innovations in how we utilize the user-generated tags, crowd sourcing, and elite coding to generate topics in a dataset characterized by both class imbalance and a large number of topics. We then use multiple steps to validate these topics. Others seeking options for identifying rare language in a large corpus of text might utilize the methods outlined here or adapt them for other approaches. Researchers interested in a qualitative evaluation of the content of political issues could also use our trained topic model as an information retrieval tool to help narrow down the large corpus of sermons for analysis.
Qualitative evaluations of the sermons suggest that pastors are willing to post sermons with content that is controversial, political, and related to current events. Yet, the results presented here are, like much of the scholarship on religious elites, based on a convenience sample. We are not confident that we understand what might prompt a pastor to post anything on SermonCentral or not, or to post a specific sermon on the website, just as we do not know fully what causes individuals to post political information on social networking websites (Barberá and Rivero Reference Barberá and Rivero2015). Future research might combine our large-n data with surveys of pastors (Calfano and Oldmixon Reference Calfano and Oldmixon2018) to understand how they see their congregation's politics or to gather additional observational data on the parishioners themselves.
What our study cannot say is the effect of these sermons on the congregants in these churches. Yet, research demonstrates that the religious content of church discussions percolate to voters' attitudes (Deckman Reference Deckman2002; Cassese and Holman Reference Cassese and Holman2017) and political attitudes shape where people attend church (Margolis Reference Margolis2018, Reference Margolis2016). Building on a robust body of experimental work (Albertson Reference Albertson2015) in the study of religion and politics (for a review, see Djupe and Smith Reference Djupe and Smith2019), research could use the material from the sermons in our dataset to develop experimental treatments with external validity to examine how they shape audience reactions. While our work helps clarify elite messaging about politics in religious environments, understanding how a religious audience reacts to these messages is an important next step.
Our work highlights the possibilities of applying new methodologies to long-standing questions in political science. The intertwined nature of religion and politics in the United States is far from a new inquiry. Yet, applying new methods and approaches opens new avenues of knowledge. That most pastors engage with political issues at least some of the time reaffirms the importance of churches as vehicles for political content, opinion formation, and socialization.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1755048320000334.
