CHARACTERISTICS SHARED BY POLITICAL METHODOLOGISTS

What are the core interests, talents, and values shared by political methodologists, that define what Leeper (Reference Leeper2018) calls the prototype of a member of the subfield? To begin answering this question, it is helpful to consult historical narratives of the genesis of the SPM (including the contribution in this issue by Mitchell and Achen). As Franklin (2008, 797–8) notes, the society was created by:

Given this genesis, we should not be surprised that the political methodology community has somewhat different priorities than other subfields of political science, priorities that focus more on how we learn about politics than what we learn about politics. Yet, the community is substantially more connected to substantive political science than someone interested solely in how evidence is constructed, such as a statistician or a philosopher of science.

Consider two idealized forms of scientific scholar: the scientist and the engineer. Scientists focus on generating new and abstract information about a topic, usually a narrow topic, without necessarily considering immediate applications or spending the time and energy required to convert an abstract idea into a practical implementation. Engineers, by contrast, tend to be less concerned with the novelty of the underlying science and more concerned with creating novel and important tools using that science. Of course, probably no real person in any field is a pure scientist or engineer as I have defined these terms: prospective applications are often in the mind of a scientist, and an engineer may need to extend abstract knowledge to make a new tool work. However, in my qualitative (but non-systematic) impression of the community acquired during years of active SPM membership,^{Footnote 1} I have observed that political methodologists bear a closer resemblance to engineers than to scientists. Moreover, the tools that methodologists build are designed to solve problems posed by substantive research in political science.

My impression is that political methodologists are mainly in the business of:

• • studying the practical problems that arise in political science research, including limitations of widespread techniques or flaws in common practices

• • bringing these problems to the attention of the community

• • developing and disseminating solutions for these problems that can be used easily by political scientists (e.g., statistical models and analytical procedures, as well as software packages to implement these models and procedures)

My impression is close to that of Mitchell and Achen (Reference Mitchell and Achen2018), who state that “[m]ethodologists identify resaerch questions and theories in political science that have encountered methodological difficulties of a statistical kind, and they develop new strategies, tools, and estimators to better answer those questions.” These tasks are closer to the problem-solving, tool-building tasks of an engineer. As King (Reference King1991, 8) notes, it is not enough to simply import ideas from other fields and apply them directly to political problems: “[m]ethodologies are not always universally applicable; they must be adapted to specific contexts and issues if data are to be put to good use.” The mission of tailoring analytical tools to the contexts and issues interesting to political scientists is a core part of the research program of methodologists.

The three tasks of methodologists I identified previously are similar to those defined by Box-Steffensmeier, Brady, and Collier (Reference Box-Steffensmeier, Brady, Collier, Box-Steffensmeier, Brady and Collier2008, 3). In their introductory chapter to the Oxford Handbook of Political Methodology, they describe how a methodologist would approach the task of studying the substantive political topic of “revolution” as follows:

Their description of methodology closely mixes substantive and technical concerns; that is, the contributions of the methodologist are oriented consciously toward the requirements and needs of substantive research and those who perform it. The desire to closely pattern the work of methodologists against the particular needs of substantive political science was present at the creation of the community. Franklin (2008, 797) notes that “SPM set as its goal the development of a political methodology devoted to questions arising from problems unique to political data rather than borrowing solutions from other disciplines whose concerns might only tangentially reflect the concerns of political scientists.” In a similar vein, King (1991, 1) emphasizes that methodology should not be simply a branch of statistics, but rather closely aligned with the substantive questions that interest political scientists and the methodological problems they encounter when they study those questions:

As a consequence of the engineering orientation of the subfield, it is my observation that political methodologists tend to be more knowledgeable than statisticians about programming, research design, epistemology, and the substance of politics. It is informative that two of the major statistical software companies (i.e., StataCorp and SPSS, Inc.) were founded (and their first software versions written) by social scientists.^{Footnote 2} Similarly, I have observed that methodologists tend to be less knowledgeable than statisticians about measure theory, proof construction, and abstract mathematics compared to statisticians. Mastery of these ideas is not as important for the engineering tasks of tool building and application when compared to other skills.

TEXT ANALYSIS OF POLITICAL ANALYSIS ABSTRACTS

We can study the scholarly priorities and values of the methodology community more systematically by examining work published in Political Analysis, “the official journal of the Society for Political Methodology and the Political Methodology Section of the American Political Science Association” (Cambridge University Press 2017a). As the only peer-reviewed academic journal formally associated with the SPM and with a mission to “publish peer-reviewed articles and letters that provide original and significant advances in the general area of political methodology” (Cambridge University Press 2017b), it is plausible to assume that the journal’s content reflects the values and priorities of the community that it serves.

I created an original data set of the abstract text for all 276 articles published in Political Analysis between 2007 and 2016. After initial data cleaning and processing,^{Footnote 3} I converted all words to stems^{Footnote 4} and then constructed a term-document matrix to determine the most frequently occurring terms in the corpus of abstracts. To create a more readable output, I replaced the stems with their most frequently occurring variant. Finally, I used the wordcloud library (Fellows Reference Fellows2014) to create a word cloud with the 100 most frequently occurring words in the corpus (figure 1). The size and shading of a word is proportional to its frequency of appearance, with larger and darker words appearing more frequently.

Figure 1 Word Cloud of the 100 Most Frequently Occurring Words in Political Analysis Abstracts, 2007–2016

The content of Political Analysis abstracts supports the view that the political methodology community values building, assessing, and improving quantitative empirical models designed to measure, explain, and predict political phenomena. The most frequently occurring stem word in the corpus is model, which appears 453 times in the data set. Of the 100 most frequently occurring words in the corpus of abstracts, at least 15 suggest a mathematical, statistical, or quantitative approach.^{Footnote 5} At least 17 words suggest the assessment of a model or method’s performance, including comparative assessment.^{Footnote 6} At least 18 words indicate an interest in measurement, causal explanation, or prediction.^{Footnote 7}

A distinction between political methodologists and statisticians residing in mathematics, statistics, or computational science departments also is suggested by this analysis. Some words that do not frequently occur in Political Analysis abstracts—including theorem, algorithm, asymptotic, consistency, likelihood, and eigenvalue—appear in titles of articles in the 2017 volume of Annals of Statistics.^{Footnote 8} Certainly, proofs and algorithms are created and used by political methodologists. But the focus of work in the methodology community is on adapting and applying quantitative methods to the substantive problems of political scientists, not on devising new abstract statistical theory. Conversely, seven of the 100 most frequently occurring words in Political Analysis abstracts (e.g., election and parties) specifically pertain to politics and, particularly, electoral politics.^{Footnote 9}

UNIQUELY METHODOLOGICAL WORDS: COMPARING POLITICAL ANALYSIS ABSTRACTS TO AMERICAN POLITICAL SCIENCE REVIEW ABSTRACTS

More can be learned by comparing the representation of words appearing in Political Analysis abstracts to those in general interest political science journal abstracts during the same period. This analysis enables us to distinguish those words that are peculiar to methodologists from those common to all political scientists. Therefore, I analyze an original data set^{Footnote 10} containing all 460 abstracts from the 2007–2016 volumes of American Political Science Review. I cleaned and processed this data, combined it into a single corpus with the Political Analysis abstract text (with a variable identifying the source of each abstract), reduced the words to stems, analyzed the text via creation of a term-document matrix, and then de-stemmed words using the most frequently occurring variant among Political Analysis abstracts.^{Footnote 11} Rather than study overall word frequency (see figure 1), for this analysis I determine the ratio of how often a word occurs in Political Analysis abstracts relative to how often it occurs in American Political Science Review abstracts.^{Footnote 12} The result is shown in figure 2; larger and darker words occur comparatively more frequently in Political Analysis. Figure 2 strongly supports the conclusions drawn from figure 1: that is, methodologists are likely to discuss and assess quantitative approaches to modeling, measurement, and causal inference with a focus on application and practice. Words such as simulations, monte carlo, and Bayesian are even more prominent than in figure 1, but words likelihood and asymptotic are still absent.

Figure 2 Word Cloud of the 100 Words with the Most Disproportionate Representation in Political Analysis Abstracts Compared to American Political Science Review Abstracts, 2007–2016

HOW DOES ONE BECOME AN ACTIVE MEMBER OF THE METHODS COMMUNITY?

How does a political scientist with the interests and skills identified in the prior section of this paper become an active member of the methods community? At a certain point in the discipline’s history, there was no such community and few people^{Footnote 13} were recognized as methodologists:

Yet, approximately 200 people currently attend the annual POLMETH meetings, and Political Analysis is the journal with the fifth-highest impact factor in political science, slightly more than American Political Science Review (Thomson Reuters Corporation 2017). There was a point at which a substantial number of people who were primarily identified with another subfield of political science became methodologists, both to themselves and in the eyes of the wider discipline.

In reading several histories of the SPM (Beck Reference Beck2000; Franklin Reference Franklin, Box-Steffensmeier, Brady and Collier2008; Jackson Reference Jackson2012; Mitchell and Achen Reference Mitchell and Achen2018), it is apparent that the founders of the subfield—those who initially defined what it meant to be a methodologist—were people who wanted to:

• • think and write about improving inference in political science in journals that prioritize discussion of methodology

• • regularly attend and present research at conferences that focus on and prioritize the discussion of methodological issues to exchange ideas with other people interested in inference in political science

• • teach undergraduate- and PhD-level courses in research design and applied statistics

• • stay informed about new developments not only in political methodology but also in the allied fields of statistics, econometrics, computer science and applied mathematics, and other areas with the aim of fruitfully adapting these methods to the problems of political scientists

• • serve in the various capacities that sustain the functioning of the SPM

Doing these things distinguished the initial members of the subfield as members of the methodology community. Although much has changed since the founding of the SPM, I agree with Leeper (Reference Leeper2018) that they continue to define active membership in the methodology community. For example, being an active methodologist in the current era means publishing work in Political Analysis and the Workshop of the American Journal of Political Science as well as Political Science Research and Methods and similar journals. It means regularly attending and presenting new reseach at POLMETH and regional conferences that share its mission, such as the St. Louis Area Methods Meetings and the Northeast Political Methodology Meeting. Therefore, to be a political methodologist, I argue that one must do (at least some of) the five things on this list.

Moreover, one first must do the things that enable a person to do these things later. As I discussed previously, a person presumably wants to (for example) publish in Political Analysis because they share the interests and values of the community it serves, and that person is able to do so because they have acquired the community’s shared skill set. It is likely that one does not simply publish an article in Political Analysis ^{Footnote 14} without first acquiring the necessary mathematical and statistical background^{Footnote 15} and reading deeply into the related methodological literature. A good (and probably necessary) beginning in joining the community is to regularly attend the POLMETH meetings: to paraphrase Woody Allen,^{Footnote 16} 80% of being a methodologist is just showing up. It turns out that “just showing up” is surprisingly rare: as Achen and Mitchell report in this issue—based on work by Dion and Mitchell (Reference Dion and Mitchell2012)—historically, most attendees of the POLMETH conference are not repeat attendees (i.e., only 37.5% of men and 44.4% of women attended the conference at least twice between 1992 and 2010).

IDENTITY, GROUP DYNAMICS, AND DESCRIPTIVE REPRESENTATION AMONG POLITICAL METHODOLOGISTS

In the introduction, I noted a distinction between sharing the interests, talents, and values of methodologists and being an active member of the methods community. A particularly visible group of scholars who share many characteristics of political methodologists but many of whom are less active in the SPM/POLMETH–centered methodology community are attendees of the women-only Visions in Methodology (VIM) conference (Visions in Methodology 2017). There are several reasons why many women who are apparently a good fit with POLMETH and the SPM choose to limit their active involvement with that community (Shannon Reference Shannon2014). A possible explanation raised in a recent article by Shames and Wise (2017, 819–20) is particularly troubling:

This argument is challenging to pin down because it appears to mix three issues that are conceptually distinct:

• • the competitive or argumentative tone of discussions at POLMETH and among methodologists generally

• • the focus on studying quantitative rather than qualitative methods by those within the methods community

• • the idea of primarily focusing on identifying and solving methodological problems in political research instead of matters of “substance,” presumably including developing theories and drawing conclusions about political behavior and policy

The following section addresses each aspect of this argument.

THE CULTURE OF THE SPM AND POLMETH

The competitive and argumentative tone of POLMETH was present from the beginning: Franklin (2008, 810) characterizes the environment at the 1984 conference as “intense,” complete with “trenchant” critique and “blistering criticism.” Although my experiences at POLMETH seem more tame than the recollections of senior colleagues, I think that the spirit of critical exchange is alive and well.^{Footnote 17} By contrast, VIM was created in part to “present research in a friendly, positive environment” (Visions in Methodology 2017).

The opportunity to receive honest, direct, and useful criticism from a discussant (who typically has 15–20 minutes to make their own presentation) and the audience (typically allotted about 30 minutes for Q&A) is one feature that I most value about presenting at POLMETH and being in the political methodology community more generally. Such feedback provides the opportunity to seriously improve one’s work before encountering anonymous reviewers. Although I have received useful feedback at other conferences, my experience is that POLMETH is exceptional in this regard. It is normal for new attendees at POLMETH to take time to acclimate to both the critical environment and being a new arrival in a relatively small and tightly knit community. Awkwardness is common, but it dissipated over time for me.

Unfortunately, matters sometimes progress beyond awkwardness. Shames and Wise (2017, 816) share a relevant anecdote from the 2014 POLMETH at the University of Georgia:

I was a presenter on that plenary panel and, net of a few details, my memory of this event is similar. My recollection is that the experience was unpleasant to witness—not because the exchange was contentious but instead because the nature of the exchange seemed to short-circuit rather than enable critical discussion. I did not learn anything from this experience about how well PhD students without a mathematical background can assimilate and apply quantitative methods, and I am concerned about what newcomers to POLMETH thought about methodologists after that exchange.

I do not believe that this anecdote encapsulates anything about the core values and priorities of the community. I also reject any idea that it is necessary to abandon a focused and critical exchange of ideas to avoid events like the one described in the anecdote. However, this story provides an explanation for why some scholars who share the core interests and values of methodologists choose not to remain active within the SPM/POLMETH–centered community. Although I think that response is reasonable, my view is that choosing to engage with the community and voice these concerns is more likely to make the community welcoming to newcomers—while also preserving its tradition of honest critique and its focus on inference—compared to choosing to disengage.

THE QUANTITATIVE AND EPISTEMOLOGICAL FOCUS OF METHODOLOGY

What about the second and third aspects of Shames and Wise’s (Reference Shames and Wise2017) argument: is quantitative and methodological research inimical to women per se? As I discussed previously, building, assessing, and improving quantitative empirical models relevant to political research are core goals for a political methodologist.^{Footnote 18}

I believe that a comparison with statistics—a discipline with an even more abstract and quantitative focus than the political methodology subfield—undermines the argument that being primarily concerned with methods and inference tends to reinforce gender disparities. If it is the case that focusing intensely on the development of new quantitative models for the analysis of substantive problems is in and of itself off-putting to women, then we should see similar levels and changes in the representation of women in statistics as in political methodology. However, statistics as a field is more descriptively representative of women than political methodology. Moreover, statistics departments have achieved impressive gains in gender diversity during the past 25 years, whereas female representation at POLMETH has remained almost the same during the same period. Even mathematics departments, which generally are even more abstract and less concerned with practical application compared to statistics, have nearly doubled their proportion of female full-time faculty during this period. Consider figure 3, which combines evidence from two sources.

Figure 3 Female Representation among Full-Time Doctoral Statistics Faculty, Full-Time Doctoral Mathematics Faculty (Excluding Departments of Statistics and Applied Math), and POLMETH Paper Authors

The first source of data in figure 3 is an annual American Mathematical Society survey of faculty in the mathematical sciences, including data on the proportion of female full-time faculty holding a PhD in “US departments (or programs) of statistics, biostatistics, and biometrics reporting a doctoral program,” as well as the proportion of female full-time faculty holding a PhD in “doctoral-granting departments of mathematics” not including departments of statistics, applied mathematics, or operations research (Loftsgaarden, Maxwell, and Priestley Reference Loftsgaarden, Maxwell and Priestley2001, 828). Figure 3 reports these proportions for all surveys conducted between 1990 and 2015.^{Footnote 19}

The second source of data is a study conducted by Dion and Mitchell (Reference Dion and Mitchell2012), whose results were published in the Washington Post (Mitchell Reference Mitchell2013) and also are featured by Mitchell and Achen (Reference Mitchell and Achen2018) in this symposium. Dion and Mitchell coded the proportion of female authors of papers presented at POLMETH relative to the total number of authors for all presented papers in the program. Although the data were presented only in graphic form, I was able to convert the data to a numerical form using WebPlotDigitizer (Rohatgi Reference Rohatgi2017). Furthermore, although the data were available only until 2012; I accessed the websites of the 2013–2015 POLMETH conferences to manually update the data.^{Footnote 20} It is obvious that not all faculty members who identify as methodologists present papers at POLMETH. However, POLMETH restricts panel presentations to papers with faculty members as primary authors (although graduate students may be coauthors). Moreover, given that I defined the methodology community as being organized principally around the POLMETH conference (and the SPM), the population of presenters at POLMETH is a prima facie valid sampling frame in which to measure characteristics of the target community.

Figure 3 shows that in statistics departments, the proportion of full-time faculty with a PhD who are women increased steadily between 1990 and 2015. Regression analysis shows that, on average, this proportion increased by 0.88 percentage points per year during this period (p<0.001, two-tailed). A similar result holds for full-time mathematics professors, in which the proportion of women increased by 0.50 percentage points per year (p<0.001, two-tailed). By contrast, the proportion of women authors of papers presented at POLMETH increased much more slowly between 1990 and 2015. Regression analysis shows that the proportion of women authors increased, on average, by 0.17 percentage points per year, a growth rate statistically indistinguishable from zero (p=0.260, two-tailed). Moreover, the 0.71-percentage-point difference between the growth rates of the proportion of women among full-time statistics faculty and the growth rate of female representation among authors of POLMETH papers is statistically significant (p<0.001, two-tailed). The 0.33-percentage-point difference in growth rates between full-time mathematics faculty and POLMETH authors also is statistically significant (p=0.031, two-tailed).

There are many possible reasons why statistics and mathematics departments have been able to more than double the proportion of full-time doctoral faculty who are women between 1990 and 2015, whereas POLMETH has not been able to achieve similar gains in the proportion of women authors of presented papers.^{Footnote 21} For example, it is possible that mathematics departments tend to disproportionately recruit women who have interest and talent in math; these women then gravitate toward more substantively applied and collaborative specializations (e.g., statistics) rather than “pure” mathematics (Schulte Reference Schulte2014). By contrast, students who enter a political science graduate program typically are not interested in math; therefore, women who gravitate toward substantive specializations do not become methodologists.^{Footnote 22} The representation of women in mathematics departments is considerably lower compared to statistics departments, which is consistent with this argument. However, the growth of women’s representation among faculty members in mathematics departments is more than twice as fast compared to the growth rate among authors of POLMETH papers. It may be that mathematics and statistics departments are making more strenuous or more effective efforts to recruit women compared to the political methodology subfield.

It is not necessary to resolve the question of why we see the results in figure 3 to rule out one important possibility: that a primary focus on inference and quantitative model development is a barrier to increased gender diversity in the field. Statistics departments evidently achieved substantially closer gender parity on their faculty than political methodologists, despite being a field at least as focused on abstract model building and assessment. Even mathematics departments, which are still far from gender parity, have increased women’s representation on faculty much faster than political methodologists have increased representation among female authors featured at our annual conference. As a result, I believe it is reasonable to conclude that POLMETH and the SPM can improve on descriptive representation without surrendering this core aspect of their identity.