Terra incognita: the role of politics and politically sensitive policy choices in sovereign rating decisions

Despite two decades of ever-intensifying scholarly interest in sovereign credit ratings, it remains incompletely understood to what extent CRAs’ assessment of the creditworthiness of countries goes beyond the analysis of macroeconomic and fiscal indicators to scrutinise politics and politically sensitive policy choices. Monitoring headline macroeconomic and fiscal indicators (like Gross Domestic Product (GDP), GDP/capita, growth, unemployment, inflation, trade balance, public debt or deficit) has direct and unambiguous relevance for assessing a government’s ability and willingness to pay because it allows for gauging the trajectory and current affordability of debt and the inclination of a government to prioritise fiscal balance over other policy goals. In contrast, the scrutiny of politically charged policy choices (like social policy, taxation, regulation, industrial policy, public investment, etc.) and political developments (such as electoral shifts, interest-group activity, or intragovernmental tensions) is a much less self-evident dimension of judging creditworthiness because the effect of such factors on a country’s ability and willingness to pay is indirect, ambiguous and often only arises in the distant and uncertain future.

Politically sensitive policy areas – like the size and focus of social spending, the types and size of taxation, the regulation of product, labour and financial markets, subsidies, or public investment projects – are often referred to as supply-side policy because they significantly influence human- and physical capital accumulation, the availability and quality of the labour force, innovation and personal incentives (Feldstein Reference Feldstein1986; Mosley Reference Mosley2000). Since these policies affect the economy’s long-term productive potential as well as the government’s future spending and revenue streams, they arguably affect long-term credit quality. Following a similar logic, political developments can be linked to future creditworthiness if one expects certain political developments to be reliably associated with certain fiscal, economic and supply-side policy choices. For example, the ideological composition of a new government is sometimes taken to be an indicator of what policy agenda will be pursued, while the strength of unions or lobbies is often seen to limit policy choice. Yet, drawing firm conclusions about future creditworthiness from current policy choices and political developments is a controversial practice. In light of the intense and unabating scholarly and ideological controversy about what concrete supply-side policy choices yield better economic and fiscal outcomes (e.g. Feldstein Reference Feldstein1986; Lucas Reference Lucas1990; Easterly and Rebelo Reference Easterly and Rebelo1993; Engen and Skinner Reference Engen and Skinner1996; Carlin and Soskice Reference Carlin and Soskice2009) and the uncertain correlation between political factors and certain policy choices (Cusack Reference Cusack1999; Ross Reference Ross2000; Potrafke Reference Potrafke2009), judging policy choices and political developments (even from the perspective of their effects on future credit quality) is an inherently value-laden, ideologically loaded and contentious exercise, which implies taking sides on politically sensitive issues at the very heart of democratic contestation.

CRAs face conflicting incentives when it comes to scrutinising policies and political developments. The potential advantages of such analysis in terms of additional information to predict the future might be more than counterbalanced by risks to CRAs’ authority from taking sides in contentious and ideologically loaded political and policy debates. Ratings need to provide forward-looking indicators of credit quality (Carruthers Reference Carruthers2013). Since current fiscal and macroeconomic performance only conveys information about current governments’ current willingness and ability to pay, rating agencies might use political and policy analysis to bolster their ability to credibly predict future fiscal and macroeconomic performance and, thus, future credit quality. However, judging politics and policy might be risky for CRAs’ epistemic authority, which the rating business is founded on. Epistemic authority guarantees the trustworthiness of the risk assessment ratings represent (Sinclair Reference Sinclair2005). Since incorporating the scrutiny of politics and policy into rating decisions involves a number of contentious, normative and ideologically loaded assumptions about how politics and policies affect the economy and fiscal performance, entering this terrain could be risky for the epistemic authority needed to undergird the credibility of ratings. This leads some scholars to expect CRAs to steer clear of such issues and focus strictly on relatively uncontroversial quantitative analysis of economic and fiscal indicators, in order to shroud their inherently subjective judgements in the “objectifying cloak of economic and financial analysis” (Sinclair Reference Sinclair2005, p. 35; Paudyn Reference Paudyn2013).

A possible way to reconcile the contradicting requirements of needing to avoid controversy that could undermine the epistemic authority of ratings and the need to integrate additional information on which forward-looking projections for the future could be based is to conform closely to market conventions and widely accepted mental models in assessing the tenuous links of politics and supply-side policies to creditworthiness. Indeed, scholars have argued that CRAs retain their authority by converging closely on market sentiment, going as far as claiming that ratings “codify what the market already knows” (Abdelal and Blyth Reference Abdelal, Blyth, Cooley and Snyder2015, p. 40, see also Sinclair Reference Sinclair2005).

However, what this implies concretely for CRAs’ approach towards political and policy analysis is somewhat obscured by the ambiguity about the question to what extent markets themselves monitor politics and policy. This issue is at the heart of the “golden straitjacket” debate. Some scholars assume that markets pay close attention to – and reward and penalise – politics and policy (Garrett and Lange Reference Garrett and Lange1991; Strange Reference Strange1996; Rodrik 2000 and Reference Rodrik2011; Streeck Reference Streeck2014). Others argue that investors use shortcuts and heuristics to economise on information costs, and therefore, they employ different levels of political and policy scrutiny to different country categories and country “clubs” (like trade alliances or other forms of inter- and supranational integration) associated with different levels of perceived political, economic and fiscal uncertainty. In developed countries and members of “clubs” associated with low levels of uncertainty, like the EU, politics and policy receive little attention, whereas developing countries and countries belonging to less prestigious “clubs” are subject to extensive political and policy scrutiny (Mosley 2000 and Reference Mosley2005; Gray 2009 and Reference Gray2013; Brooks, Cunha and Mosley Reference Brooks, Cunha and Mosley2015).

This view of information-economising investor behaviour stands in marked contrast with conceptions of more sophisticated decision-making by investors. For example, Breen and McMenamin (Reference Breen and McMenamin2013) and Sattler (Reference Sattler2013) contend that market reactions to political developments (like the outcomes of elections) are modulated by the political institutions (specifically the number of veto points) in a country. Such nuanced assessment of the consequences of conspicuous political events presupposes intimate knowledge of domestic political relationships by investors. Given this theoretical ambiguity, it is difficult to determine how CRAs should be expected to approach political and policy analysis, if their goal is to replicate the mental models of markets. Whether they only monitor politics and policy closely in categories and “clubs” associated with greater uncertainty, or they engage in nuanced analysis of these issues across the board, is an issue to be empirically decided.

Empirically, the existing evidence is too scarce to draw definitive conclusions. Quantitative studies have mostly focused on confirming the importance of fiscal and macroeconomic indicators for credit rating scores (Cantor and Packer Reference Cantor and Packer1996; Afonso Reference Afonso2003; Afonso et al. Reference Afonso, Gomes and Rother2007), and insofar as they incorporated political factors, they attempted to capture the impact of the broadest political institutional frameworks such as regime type or of major political upheavals like coups and revolutions (Haque et al. Reference Haque, Mathieson and Mark1998; Archer et al. Reference Archer, Biglaiser and DeRouen2007; Beaulieu et al. Reference Beaulieu, Cox and Saeigh2012; Biglaiser and Staats Reference Biglaiser and Staats2012). The effects of day-to-day politics and policy have been less investigated, although two studies provide evidence that elections and the partisan colour of governments affect sovereign ratings (Vaaler et al. Reference Vaaler, Schrage and Block2006; Barta and Johnston Reference Barta and Johnston2018), while a third shows that ratings react negatively to generous entitlement systems but are neutral towards other types of social policies (Barta and Johnston Reference Barta and Johnston2019).

Qualitatively oriented studies have emphasised how difficult it is to glean unambiguous information from CRAs themselves about the role of politics and policy in their decisions about ratings. Archer et al. (Reference Archer, Biglaiser and DeRouen2007, p. 357) report major obstacles in obtaining definite answers about the role of politics in rating decisions from interviews with sovereign rating analysts. In a similar vein, Bruner and Abdelal (Reference Bruner and Abdelal2005, pp. 199–200) point out that even though CRAs publish detailed methodologies to explain the thought processes that underlie rating decisions, these documents discuss policy at a level of such generality that it is impossible to dispel the ambiguities that surround what is essentially a highly subjective assessment process and to fully understand to what extent and how rating decisions incorporate policy developments.

In the decade since the publication of these studies, CRAs have only become more taciturn about the standards along which they assess the impact of politics and policy choices on creditworthiness. Although sovereign rating methodologies have been significantly expanded and revised in the wake of the global financial crisis and the European sovereign debt crises (as CRAs sought to restore their credibility by increasing the transparency of their practices), recent editions reveal even less about CRAs’ attitude towards day-to-day politics and policy than before. Whether out of a desire to maintain epistemic authority by avoiding mention of contentious assumptions about the long-term impact of political factors and policy on the economy and the budget, or because CRAs in fact have little interest in day-to-day politics and in policy, the newest editions of the sovereign methodologies eschew these issues altogether. This stands in marked contrast with the extensive discussion on the importance of lasting economic and monetary institutions and current macroeconomic and fiscal outcomes. Potentially politically sensitive policy areas (like welfare arrangements, the size of the public sector, the characteristics of tax systems, regulatory regimes, etc.) are not even mentioned.Footnote ³ Nor do the methodologies comment on the effect of elections or the partisan colour or structure of the government in power on the assessment of sovereign risk.

In contrast to their reticence about their general principles of assessing politics and policy, however, CRAs are remarkably forthcoming about the factors that influence their specific rating decisions. Each time a CRA officially announces its decision to raise, lower or affirm a country’s rating and to issue negative, positive or stable outlooks about future rating changes, a report is published in which the CRA spells out the reasons that motivated its decision.Footnote ⁴ These reports usually range between 300 and 600 words, and their frequency depends on how often a CRA deems it necessary to revisit a country’s rating. They are usually published in their entirety by outlets like Reuters or Bloomberg, whereas other financial news sources quote lengthy excerpts in their reporting of new rating decisions. These documents provide a rich source of information about how CRAs arrive at, justify and communicate their decisions, but no one has systematically analysed them to draw conclusions about the role of politics and policy choices in influencing ratings.

The stark contrast between the secrecy surrounding general principles and the openness about specific cases in rating reports is consistent with the tension, explored above, between CRAs’ incentives to avoid taking an explicit stance in theoretical political or policy controversies and their need to bolster the credibility of their forward-looking assessment of credit quality. Rating reports make it possible for CRAs to inspire trust in their decisions by elaborating on all the factors that influenced the final verdict. At the same time, CRAs can eschew taking sides in abstract theoretical debates through high-profile blanket pronouncements about what policies and politics generate optimal results for creditworthiness in general, which could damage their epistemic authority. Arguably, comments on a concrete country’s concrete policy choice or political developments are less likely to stir a highly publicised controversy than making generalised statements in defining documents like rating methodologies.

Anecdotal evidence gleaned from reading a random sample of reports suggests that CRAs complement the assessment of economic and fiscal trends with scrutiny of domestic politics and policy. The quotations below provide an assortment of illustrative examples of political and policy analysis from reports from each of the “big three” agencies, about a broad range of countries, justifying negative, neutral and positive rating decisions:

Evincing the desire to provide forward-looking assessments of risk, the reports also spell out possible future political and policy scenarios to alert investors to possible rating changes (and, in the process, also signalling to governments the penalties and rewards associated with different political and policy choices):

These excerpts indicate that CRAs analyse politics and supply-side policy in AAA-rated Germany just as much in-depth as in (at the time) B-rated Venezuela and (at the time) CCC-rated Ukraine. They also suggest that commentary on policy is, at least occasionally, informed by the neoliberal paradigm, demanding, for example, the privatisation of pension funds, independent central banks, trade liberalisation (in Venezuela), labour market liberalisation and the reform of tax and welfare arrangements (in Germany), the reduction of the tax burden and health care and pension reform (in the Ukraine). Casual further reading of more than 100 reports reinforces this impression, with reports regularly talking about the need for structural reforms in labour markets, welfare and health in ways that are consistent with a neoliberal view of policy. Interestingly, reform also features repeatedly in discussions of politics (e.g. the need to reform the federal system in Germany in reports from the early 2000s, or the need for constitutional reform in Italy in the first half of the 2010s), besides discussions of day-to-day politics.

The excerpts also document complementarity between political and policy analysis as CRAs seek to provide projections about future economic and fiscal developments. Policy is scrutinised in the context of assessing the longer term economic and fiscal health of a country, whereas political analysis is employed to predict potential changes in policy. Thus, these excerpts suggest that policy analysis and political scrutiny represent complementary components of CRAs’ attempts to reduce uncertainty about future macroeconomic and fiscal performance.

The rest of this article systematically explores how the assessment of politics and policy features in CRAs’ rating reports. We entertain the possibility that categories and “clubs” of countries associated with different levels of uncertainty are treated differently by CRAs, mirroring the behaviour of investors documented by Mosley and others. Earlier versions of CRAs’ methodologies provided indication that CRAs associate certain categories and “clubs” of countries with lower levels of (political and policy) uncertainty and, therefore, treat them differently. Fitch stated that developed sovereigns constitute a privileged category (Fitch Ratings 2009, p. 5), whereas Moody’s and S&P confirmed that prestigious “clubs” like the EU are seen more favourably when creditworthiness is evaluated (Moody’s 2008, S&P 2006, p. 7). Such statements have been removed from more recent editions of the methodologies issued after the global financial and economic crisis (S&P 2011, 2013 and 2016, Moody’s 2016, Fitch Ratings 2016), suggesting that the crisis might have changed CRAs’ perception of lower uncertainty in developed sovereigns and members of certain “clubs.”

Therefore, we test five related hypotheses about the variation in the weight of political and policy scrutiny in CRAs’ assessment of creditworthiness across countries and time. We form separate hypotheses about political and policy scrutiny, in case they might be applied differently across country groups associated with different levels of uncertainty. We also form separate hypotheses about country categories (e.g. developed/emerging) and membership in “clubs” (like the EU) because they represent slightly different theoretical propositions. Finally, we hypothesise that the global financial and economic crisis represents a trend break in CRAs’ approach to uncertainty. The crisis represented a major shock to CRAs’ authority, which led them to re-evaluate some of their theoretical propositions (which is reflected in major revisions in all of the three agencies’ methodologies), while the series of bank failures and sovereign debt crises in advanced economies also called attention to the mistakenness of markets’ earlier complacency about the creditworthiness of developed sovereigns. Therefore, we expect the weight of political and policy scrutiny across countries to change after the crisis. On the one hand, the perception of lower uncertainty in developed countries might disappear. On the other hand, the relative weight of political and policy scrutiny might change if a generalised sense of elevated uncertainty causes CRAs to move towards political analysis in an effort to better predict changes in policy and in economic and fiscal performance.

• Hypothesis 1a (politics and categories): The relative weight of political commentary is significantly heavier in emerging economies than in developed countries.

• Hypothesis 1b (politics and “clubs”): Within the emerging category, the relative weight of political commentary is significantly lighter in countries that belong to organisations (“clubs”) associated with positive connotations, like the EU.

• Hypothesis 2a (policy and categories): The relative weight of policy commentary is significantly heavier in emerging economies than in developed countries.

• Hypothesis 2b (policy and “clubs”): Within the emerging category, the relative weight of policy commentary is significantly lighter in countries that belong to organisations (“clubs”) associated with positive connotations, like the EU.

• Hypothesis 3 (trend break): The financial crisis of 2008 causes a trend break in the pattern of political and policy commentary, affecting the relative emphasis on politics and policy, or the treatment of different country groups, or both.

The empirical strategy: interrogating country rating reports about the relative importance of politics and policy in rating decisions

Our empirical investigation of the above hypotheses is based on a quantitative text analysis of 635 credit rating reports issued by Standard & Poor’s Financial Services LLC between 1 January 1999 and 30 June 2012 for European countries, broadly understood. We gain a better understanding of the weight that politics and policy receive relative to the economy and fiscal performance in rating reports by measuring the relative frequency of words associated with each topic. We investigate the patterns of variation across country groupings and time periods by using linear mixed-effects models fitted with residual maximum likelihood.

The quantitative text analysis

To explore the relative emphasis S&P places on politics, policy, the economy and fiscal conditions in its assessments of creditworthiness and its official discourse about its decisions, we analyse the relative frequency of terms associated with these four issue areas in the rating reports. We derived the word frequencies in three steps. First, we generated a list of words – excluding stop words – from all of the rating reports using the software “tm: Text Mining Package in R” (Feinerer and Hornik Reference Feinerer and Hornik2017). We did not use word stemming to ensure that we could interpret the terms correctly for the qualitative coding and we kept the top 97% of words according to frequency. Dropping the least frequent 3% of words from our database is appropriate in quantitative text analysis to exclude outliers (Benoit and Herzog Reference Benoit, Herzog, Bachner, Ginsberg and Wagner Hill2017, p. 146). At the same time, it leads to the exclusion of country-specific words like proper nouns such as names of political personalities, policy makers and political or policy-making institutions, which (as the quotations in the previous section show) repeatedly feature in reports, but (due to their specificity) do not exceed the 3% threshold in the joint sample of 40 countries over 14 years. Given that the use of such proper nouns likely suggests political or policy commentary, this omission should bias our results about the weight of political and policy-related issues in rating reports slightly downwards, that is, we are likely to slightly underreport the weight of politics and policy in the rating discourse.

As a second step, we conducted two independent qualitative codings of the resulting list of words to determine whether they relate to politics, policy, the economy or fiscal developments, or they are neutral. Both coders read an extensive sample (approximately 100 random reports each) prior to coding the word list, in order to gain familiarity with the terminology applied in the reports. We reconciled the two codings where any discrepancies existed.Footnote ¹¹

We labelled words “Economic” if they can be directly linked to long- and short-term economic issues on which we know (from S&P’s methodologies as well as from sampling reports) S&P places significant emphasis. These include long-term overarching characteristics of the economy (e.g. diversification, export-orientation, GDP per capita, etc.) and indicators of short-term performance (e.g. headline macroeconomic indicators like growth, inflation, employment, trade balance or the exchange rate, but also short-term sectoral issues like banking or construction performance, household consumption, etc.). We also used our knowledge of the terminology of reports when coding terms like “resilient,” “inefficient” or “recovery” under this label. Under “Fiscal,” we coded words referring to the usual headline fiscal indicators (e.g. debt to GDP, deficits, revenues, spending, etc.) or to the financing and management of the outstanding debt (e.g. rollover, bonds, maturity, etc.). All of these aspects of fiscal policy feature prominently in S&P’s methodology. We also coded words under this label that we recognised as recurring phrases in the terminology of reports (like “sustainable trajectory,” “tightening” or “prudence”).

As previously explained, CRAs’ methodologies provide little help in pinning down the political and policy-related aspects of the rating assessment. In the absence of clear cues, we included under “Politics” words that unambiguously relate to governing (like coalition, government, minister, parliament, etc.), to interest representing organisations (e.g. agenda, party, union), electoral issues (e.g. election, referendum), and to conflict and its resolution (agreement, consensus, summit, tensions, etc.). This also reduced the concern about exclusion of proper nouns, as they would usually be coupled with terms that were counted (such as “Chavez administration” and “SPD/Green coalition”). Under “Policy,” we included terms that relate directly to a specific policy area (e.g. health, pensions), or policy actions (e.g. austerity, programme, package, reform), but our codings also relied on our knowledge of the terminology of reports surrounding politically sensitive policy issues (e.g. when classifying “challenge,” “commitment,” “credible,” “resolve” under this label). Words that did not unambiguously relate to one of the four issue areas, we coded as “Neutral.” Table A7 in the Appendix displays the list of words in each in our four coding labels (plus the residual “Neutral” label).

Our final step was counting the frequency of the coded words and recording the total word count in each report (using the same text-mining package). We work with the relative frequency of terms as a share of the total non-neutral word count for the report, rather than the absolute numbers, because reports vary greatly in length (expanding pronouncedly in the years since the global financial and economic crisis, see Figure 1). The average number of words per report is 296.6 and the standard deviation is 105.5. Using relative frequencies allows us to eliminate potential confounding effects such as different writing styles of different analysts assigned to specific countries (which could lead to shorter/longer reports, and therefore fewer/more words, in the case of specific countries) or the general lengthening of reports in the wake of the crisis. As a robustness check, we also ran our models using absolute word count of politics- and policy-related terms, including a control variable for report length. The results using absolute word count (presented in Table A3 of the Appendix) are consistent with the findings using relative word frequencies (presented in Table 2).

Figure 1. Word count per report over time by country group.

Table 1. Summary statistics of relative word frequencies (all countries, 1999–2012)

Table 2. Results of the linear mixed-effects models. The impact of country groups and timing relative to the crisis on the relative frequency of words related to politics and policy in country rating reports

***p < 0.001, **p < 0.01, *p < 0.05.

^† Country groups are developed/transitioning/emerging.

^‡ Periods are before/during/after the financial crisis. Coefficients are unstandardised. Standard errors are in parentheses. p-Values are estimated using the Satterthwaite approximation. The analysis includes 635 reports across 40 countries. The intraclass correlations for the models are 0.286 and 0.269 for politics and policy, respectively, which represents the amount of variance that is accounted for by between-country differences. Model diagnostics were checked, and residuals were normally distributed and cases of high leverage did not influence the substantive findings. ^{Footnote 13} Pseudo R-squared calculated using MuMIN (Barton Reference Barton2019).

Since using relative frequencies makes it possible that our two variables of interest (the share of political commentary and the share of comments related to supply-side policies) covary in ways that variation in one variable confounds variation in the other, we check the correlation of our four categories. We find that politics- and policy-related terms have negligible correlation with each other (r = 0.13). The strongest correlations are between the relative weight of “economic” terms and the relative weight of each of the three other codes (−0.46, −0.56 and −0.63 with “fiscal,” “politics” and “policy,” respectively), suggesting that the weight of economic analysis expands and contracts at the expense of the analysis of the other three areas, while the other three areas are relatively unrelated to each other.

Having measured the relative frequency of words in all four issue areas in all of the 635 credit rating reports, we analyse the variation in the relative frequency of terms related to politics and policy across country groupings (“developed,” “transitioning” and “emerging”) and time relative to crisis (precrisis from 1999 to 2007; during the onset of crisis in 2008 and 2009, and after the onset of the crisis from 2010 to 2012), in accordance with our hypotheses specified above. To do so, we record the country and the year associated with each rating report. Individual reports serve as our unit of analysis. Our dependent variables are the relative frequency of terms associated with politics and policy in rating reports. Our independent variables are country group, time period and the interaction of the two. Given the varying length of reports, we include the word count of the report as a control variable.

The aim of our empirical analysis is to test whether there are significant differences in the extent to which politics and policy are discussed across country groups and over time. We are also interested in the interaction of country groups and time periods. We apply linear mixed-effects models (fitted with residual maximum likelihood and random intercepts) instead of analysis of variance (ANOVA) because our sample is unbalanced, with a varying number of reports among countries and among years, which makes both time-series cross-sectional and factorial ANOVA analysis unreliable in capturing the effect of country groupings and period on our dependent variables. As Barr et al. (Reference Barr, Levy, Scheepers and Tily2013) explain, the strength of this method (in contrast with ANOVA) is in the flexibility of the model design with regard to specifying fixed and random effects, and the fact that it can accommodate an unbalanced sample. Benefits of linear mixed-effects models over factorial ANOVA also include the possibility to establish the direction and magnitude of the differences across categories.

The aim of the analysis is to measure variation in the dependent variables across categories rather than to establish complete causal explanations for the dependent variable. This is why our models do not control for the objective economic factors but focus on variation across country groups, time period and the interaction of the two. The decision to focus on comparison across these categories is also based on the limited number of country level units (40) and high covariance between objective economic factors, such as GDP-levels and country groups. Our models estimate the relative frequency of terms associated with politics and policy, treating country grouping, time period relative to the crisis and the interaction between the two as fixed effects, meaning that we fit a model with a single coefficient for these variables as we do not expect the slopes to vary across countries. The models include a random intercept for countries to account for the lack of independence between reports for the same country. We also control for the length of the report as a fixed effect in each model given that this varies, as shown in Figure 1. The regression equation used in our analysis is

$${y_{ij}} = {b_0} + {b_1}*{(country\;group)_{ij}} + {b_2}*{(time\;period)_{ij}} + {b_3}*(country\;group)*{(time\;period)_{ij}} + {b_4}*{(world\;count)_{ij}} + {v_{i0}} + {e_{ij}}$$

for i ϵ {1, …, n} and j ϵ {1, …, m}, where i represents the country and j the number of the report per country. y is the share of non-neutral terms in the report corresponding to the category being analysed. b_1–4 represents the coefficients for the fixed effects. v_i0 are the random intercepts for countries.

The analysis was done in R using lme4 and lmerTest packages (Bates et al. Reference Bates, Maechler, Bolker and Walker2015; Kuznetsova et al. Reference Kuznetsova, Brockhoff and Christensen2017). The models are fitted using restricted maximum likelihood, and p-values are estimated using the Satterthwaite approximation in the lmerTest package, which is shown to produce acceptable type 1 error rates even with limited sample size (Luke Reference Luke2017). The estimated means for categories and plots were generated using emmeans package (Lenth Reference Lenth2019).

The results: the emphasis on politics and policy in rating reports

Table 1 presents the summary statistics of the relative frequency of terms in all four issue areas as a percentage of all words coded under one of our four substantive labels for all 40 countries over the entire period between 1999 and 2012. Figure 2 presents the weight of the different issue areas over time. We excluded the residual neutral words from our analysis of the relative weight of the four issue areas to avoid confounding effects of potential changes in the share of “Neutral” terms. As these words could not be assigned theoretical meaning in the coding process, predicting their share within the reports in a model would not produce theoretically interpretable results. (However, as a robustness check, we replicate our models including “neutral” words in the total word count and the substantive findings are consistent. See Table A5 in the Appendix.)

Figure 2. Relative word frequencies per issue-area over time.

Table 1 shows that although the analysis of economic and fiscal conditions dominates S&P’s discourse on countries’ creditworthiness, terms referring to politics and policy also have notable weight. Words clearly relating to economic and fiscal performance account for an average of 45 and 22%, respectively, of the total words coded under one of our substantive labels in all the countries across the years. Terms unambiguously referring to politics and policy make up 16 and 15% of the words of the reports, respectively, accounting for a third of the non-neutral words.

Table 2 presents the results of our analysis of the variation of the weight of political and policy scrutiny across country groups (developed/transitioning/emerging) and periods (before/during/after the onset of the crisis), using linear mixed-effects models. (Results for economic and fiscal terms can be found in Table A2 in the Appendix.) It shows that country groups, time period relative to the crisis and their interaction statistically significantly influence the weight of political commentary in reports, suggesting that (1) S&P approached politics in different country groups differently; (2) the crisis increased the average weight of political scrutiny for the whole sample; and (3) the crisis diminished the differential treatment of country groups. In the case of policy commentary, we find no significant variation in the weight of policy-related words across country groups, only across time periods, with attention to policy declining after the crisis. We elaborate on these results below with the help of graphs that visualise the interaction between country group and time period by displaying the estimated marginal means with confidence intervals for ease of interpretation.

The models for politics demonstrate substantial variation in the share of political terms across country groups and over time. To interpret the main findings, we refer to Model 2, which includes the interaction between country groups and time. Our findings are robust across models that include or exclude the interaction term or use the absolute instead of relative word count (see Table A3 in the Appendix). Overall, we detect significantly more scrutiny of politics in the emerging group before the crisis (B = 0.086; p < 0.001) compared with the developed group (base category) and no significant difference between developed and transitioning (B = 0.012; p > 0.05). This suggests that on average, before the crisis, emerging countries have 8.6 percentage points more substantive words related to politics in their reports when compared with developed countries. For the time dimension, we detect an overall heightened scrutiny of politics for developed countries after the crisis (B = 0.068; p < 0.001) compared to before (base category), suggesting an average increase of 6.8 percentage points in the substantive words related to politics. The interaction terms suggest a convergence between developed and emerging group after the crisis (B = −0.113; p < 0.001). Transitioning countries do not differ from developed countries before the crisis, but the interaction term suggests that after the crisis, countries in the transitioning category have a slightly more moderate increase in political scrutiny compared to developed countries (B = – 0.040; p < 0.01)

In order to display our results more clearly, we estimate the marginal mean values for each country group and time period (as recommended by Brambor et al. Reference Brambor, Clark and Golder2006).Footnote ¹² The interactions are shown in Figure 3, which depicts the estimated mean percentage of political commentary across country groups and periods based on Model 2 in Table 2, with 95% confidence intervals. It shows that in the precrisis years, the weight of political commentary differed among developed, transitioning and emerging groups with politics receiving more emphasis in the emerging group than in either the developed or the transitioning one. The distinction between different country groups disappears with the onset of the crisis. At the start of the crisis in 2008, attention to politics plummets in the emerging group, as economics gains supreme importance in the middle of the economic and financial turmoil. After the crisis, attention to politics is heightened in all country groups and the discrepancy between the country groups disappears. The estimated mean share of political terms in the reports for developed countries increased from 14 to over 20.8%.

Figure 3. Percentage of politics terms by time period and country groups, 1999–2012 (estimated marginal means based on model 2).

Cross-country variation in the weight of political scrutiny in the precrisis years confirms our hypotheses 1a and b, in conformity with our expectation that the intensity of scrutiny varies with perceived levels of uncertainty across different categories and “clubs” of countries. Politics was scrutinised significantly more lightly in developed countries than in emerging ones, and new EU-members benefited from association with a prestigious “club,” even though their level of economic development was much closer to emerging than to developed countries in the 1990s and 2000s.

The temporal pattern also lends support to our uncertainty explanation for variation in emphasis on political analysis. Attention to politics significantly intensifies in developed countries and new EU-member states after the crisis dispels the illusion that such countries have more predictable economic and fiscal trajectories than emerging countries (which makes the differentiation between-country groups disappear). We conducted auxiliary analysis of only eurozone countries (presented in Table A6 in the Appendix) to test whether the heightened emphasis on politics in our developed group is due to increased scrutiny of “periphery” countries, referring to the core-periphery distinction that emerged during the eurozone crisis. We detect no difference between core and periphery at the onset of the crisis. In the later years, the estimates increase dramatically for both core and periphery countries, and the estimated share of political terms is actually higher in the core than the periphery by approximately 3 percentage points (see Figure A12 in the Appendix). This finding demonstrates that the escalation in political scrutiny in the developed category is not driven by increased attention to politics in the “periphery” of the eurozone but affects the entire group in an undifferentiated fashion.

Figure 4 visualises our results for average levels of policy commentary across country groups and periods. The share of policy-related terms is similar in all country categories in each period (differences are not statistically significant). The weight of policy-related words fell by 3.6 percentage points during the onset of the crisis due to the temporary surge in emphasis on economic factors as a result of the shock (B = –0.036, p < 0.001) and remained at the lower level after the crisis (B = −0.033, p < 0.001). Lack of variation in policy scrutiny across country groups disconfirms hypotheses 2a and b: S&P has never allowed countries associated with lower uncertainty “more room to move” in policy and the analysis of such policies is a standard component of the assessment of creditworthiness for any type of country. It appears that even prior to the crisis no country category or “club” was deemed to have predictable enough economic and fiscal trajectories to warrant a more relaxed attitude about policy. At the same time, policy analysis is to some extent “crowded out” by greater emphasis on political scrutiny in the context of increased uncertainty after the crisis.

Figure 4. Percentage of policy terms by time period and country groups, 1999–2012 (estimated marginal means based on model 4).