THE IMPORTANCE OF APPLICATIONS IN POLITICAL METHODOLOGY

Political methodology—the study of methods for answering political science questions—has two parts: (1) the questions in political science and (2) the methods for answering these questions. Political methodologists are fundamentally concerned with developing and characterizing methods to answer political science questions more scientifically. While some methods can be broadly applied to many areas in political science, others are tailored to answer particular political science problems. Much of political methodology lies in adapting solutions developed in different contexts to the political science realm, taking into account the data realities of a particular situation and creating tools that better fit the needs of applied researchers (King Reference King1990).

Because the purpose of political methodology lies in answering questions, it is no wonder that many political methodologists wear two hats: one focused on answering questions in their particular subfield, another on writing papers developing and characterizing methods that could be used more generally. In my own methods work, I take inspiration from questions in Chinese politics to develop approaches that can better answer these questions (Lucas et al. Reference Lucas, Nielsen, Roberts, Stewart, Storer and Tingley2013; Roberts, Stewart, and Airoldi Reference Roberts, Stewart and Airoldi2016; King, Pan, and Roberts Reference King, Pan and Roberts2014). While the field of political methodology is fortunate to have a few scholars exclusively focused on developing methods, these scholars are few and far between. The vast majority of political methodologists run into methodological problems in the course of their substantive work and then develop approaches to solve these methodological roadblocks. Political methodology is not about developing methods without reference to their implementation, it’s concerned with need-finding and creating scientific approaches to answering political science questions that frequently have difficult-to-use data.

Because political methodology applies to every question in political science, scholars in every subfield engage in academic debates over the best methods to use for particular questions. For example, in my subfield of comparative politics scholars face a particular set of questions and circumstances that require innovative approaches and therefore have long been at the forefront of developing new methods in political science. An entire cohort of comparative political scientists have developed and honed approaches for running field experiments and taking advantage of natural experiments in different countries around the world (Dunning Reference Dunning2012; Humphreys and Weinstein Reference Humphreys and Weinstein2009). Comparative politics researchers are well represented in methodological groups such as Evidence in Governance and Politics (EGAP), which is focused on experimental research. As data in comparative contexts has become more rich, political methodologists in comparative politics have devised creative ways to use quantitative methods to make inferences about difficult to reach places using new technology. Cell phone applications, satellite images, and online geo-location are only a few ways that comparativists have used technology to get better data in difficult-to-reach places (Christia et al. Reference Christia, Yao, Wittels and Leskovec2015; Pierskalla and Hollenbach Reference Pierskalla and Hollenbach2013). Text as data methods are allowing comparative political scientists to analyze more data about their countries of interest in less time (Laver, Benoit, and Garry Reference Laver, Benoit and Garry2003; Eggers and Spirling Reference Eggers and Spirling2014; Catalinac Reference Catalinac2016). While data provided by governments in many countries cannot be taken at face value, scholars have used this creatively, namely, to identify corruption and manipulation (Mebane Jr Reference Mebane2010; Ghanem and Zhang Reference Ghanem and Zhang2014; Hollyer, Rosendorff, and Vreeland Reference Hollyer, Peter Rosendorff and Vreeland2014; King, Pan, and Roberts Reference King, Pan and Roberts2013) or use data errors to measure lack of institutionalization (Golden and Picci Reference Golden and Picci2005; Lee and Zhang Reference Lee and Zhang2017). Similarly, methods developed for American politics are not always directly applicable to comparative politics, and political methodologists in comparative politics are amending methods originally developed for US applications so that they are better applied to comparative contexts (Ferree Reference Ferree2004; Adida et al. Reference Adida, Ferree, Posner and Robinson2016; Blair and Imai Reference Blair and Imai2012).

Because researchers in comparative politics are more likely to be working in data scarce environments, they have also been required to be creative in developing methods that can be useful under these conditions. Comparative politics scholars have been at the forefront of developing methods for difficult-to-study applications; for example, developing technology to do better measurement or to elicit more accurate answers to questions (Abdelal et al. Reference Abdelal, Herrera, Iain Johnston and McDermott2006), case analysis and process tracing (Collier Reference Collier2011), and pairing qualitative and quantitative methods together in mixed methods research to answer questions in environments that are less favorable to traditional statistical approaches (Dunning Reference Dunning2008; Glynn and Ichino Reference Glynn and Ichino2015; Lieberman Reference Lieberman2005; Mahoney and Thelen Reference Mahoney and Thelen2015).

MISSING METHODOLOGISTS

When one looks at the breadth of scholars in political science doing exciting methods research in even just the example of comparative politics (not to mention other subfields) it’s a wonder that more students in subfields outside of American politics don’t come to graduate school intending to do methodology. While data scarcities and implementation challenges that come hand in hand with field research can make directly applying established methods frustrating, they also create interesting opportunities to develop and apply new approaches.

Yet, few of the papers presented in the official venues of political methodology in our field are related to questions outside of American politics. Even within American politics, political methodologists tend to focus on a particular set of questions within American politics—those concerning ideology, voter turnout, and Congress and not other parts of the subfield like race and ethnicity.

With the help of undergraduate political science students, I coded the subfield applications of each of 88 methodology papers in all the conference programs available from recent POLMETH meetings from 2012–2016.^{Footnote 3} Almost 60% (52 papers) had applications in American politics. Despite the fact that comparative politics is the largest section in APSA, only a quarter of the papers at POLMETH had applications in comparative politics and only 15% in international relations. Among the 52 papers with applications in American politics, these papers focused on a narrow set of American politics applications. Indeed, there were as many papers with applications to ideology, voting, and Congress in the US as there were papers that included applications to comparative politics and international relations combined.

This pattern is also clear when examining the overlap between those in the political methodology section of APSA and other sections of APSA. Figure 1 plots the proportion of members in each section of APSA who are also members of the political methodology section. Sections that tend to include more scholars studying American politics, such as elections, public opinion, and voting behavior, have a larger proportion of their members in the political methodology section than even that of the methodology section political networks. Even though sections like the African politics conference group and race, ethnicity, and politics are conducive to quantitative approaches, they have very little overlap with the political methodology section.

Figure 1 Proportion of Members in each APSA Section Who are also in the Political Methodology Section

Where are the political methodologists in other subfields, if not at the annual political methodology meeting or in the political methodology APSA section? Partly, these methodologists are attending other methods conferences like EGAP conferences, Text as Data, and experiments conferences. It could be that because POLMETH is in the summer when students who study other countries tend to be in the field, they are less likely to attend the general political methodology meeting. Perhaps we do not do enough to press students studying questions outside of American politics to attend and participate in methodology conferences. But somehow the political methodology meeting is missing a swath people who study a broader range of political science questions.

That the political methodology meetings do not reflect the field of political science substantively is problematic not only because the general section for political methodology should serve all questions in political science, but also because the lack of substantive representation may explain why political methodology does not reflect the field of political science demographically. As has been previously pointed out elsewhere (Teele and Thelen Reference Teele and Thelen2017), the subfields that are over-represented in the Society of Political Methodology also tend to be those that have a higher proportion of men.

Figure 2 shows the proportion of women by section membership in APSA. The red dotted line shows the overall proportion of women in the entire field, 31% (Teele and Thelen Reference Teele and Thelen2017). With the lowest proportion of women of all sections in political science (21%), political methodology is at the bottom. This is consistent with attendance at the annual POLMETH meeting, where women make up only very few of presenters, as noted by others in this symposium.

Figure 2 Proportion of Women by Section at APSA

Red dotted line indicates overall average of women in political science.

Interestingly, the subject matter that is the focus of applications at POLMETH happens to be dominated by men. Sections such as legislative studies, political organizations and parties, and elections, public opinion, and voting behavior are all sections that are less representative of women overall and are more likely to be the focus of papers at the political methodology conference. On the other hand, organized sections such as comparative politics, comparative democratization, African politics, race, ethnicity and politics, women in politics, and European politics have more female members than the overall average and tend to be less well represented by papers at POLMETH.

Of course, it does not follow directly from this correlation that expanding the scope of general venues in political methodology would also encourage more women to participate. Many more factors influence women’s decisions to go into political methodology (Unkovic, Sen, and Quinn Reference Unkovic, Sen and Quinn2016; Shames and Wise Reference Shames and Wise2017). I do not want to suggest that broadening the applications at POLMETH would be the silver bullet that would completely solve the problem of including more women in political methodology.

However, regardless of whether or not increasing the scope of POLMETH would increase diversity, the narrow substantive focus of the general methodology conferences does a disservice to the field as a whole, and expanding the scope would push our community to conduct better science. Political methodology should serve the whole field, but as it stands is more likely to serve questions where data is easy to download and field work is not required. Because many mistakenly confuse methodology with the type of method, the meeting sometimes rewards the development of complicated statistical methods that can often only narrowly be applied, at the cost of exploring new applications or simpler approaches that would have a broader impact on research in political science.

Evidence from other methods conferences and APSA sections also suggests that women are indeed interested in and working on methodology in the context of different political science questions, even if they are not attending POLMETH. Not only is the qualitative methods section of APSA more representative of women, the experiments section of APSA is also representative of women in the field, as is the political networks section, both of which are focused on the use of quantitative methods in political science. This is further reflected by the EGAP group, which is focused on experimental research that includes applications to many countries outside of the United States and is nearly half women.

In 2016 one of the Society for Political Methodology’s own conferences for women, Visions in Methodology, received around as many proposal applications as the annual POLMETH conference, despite having about a five times more competitive acceptance rate. My RAs and I took the same approach to coding the papers presented at VIM in the available programs on the website. VIM papers reflected a much broader base of political science research—only 43% of papers has applications in American politics, a full 36% of papers had applications in comparative politics and 18% of papers in international relations.

STEPPING UP TO THE CHALLENGE

There are many people across subfields in political science who are working on methods, but they are not attending the general political methodology meeting and are not participating in the APSA political methodology section. They represent a more diverse group of political scientists, both in terms of the questions they focus on and in terms of their demographics. Their participation in meetings such as POLMETH would be beneficial to the field, as it would transmit methods across subfields and deepen the development of methods in all areas of political science.

How can we attract political methodologists who are focused on a broad range of applications? One way is to widen our networks beyond our substantive fields and encourage people working on methods in areas of political science that are underrepresented at POLMETH to attend the meeting. This could mean partnering with a comparativist to include applications to both comparative and American politics in your next POLMETH methods paper or encouraging someone developing methods in other subfields to attend the POLMETH meeting.

Making POLMETH inclusive of all subfields in political science may also mean changing its structure. Scholars often tend to first attend POLMETH in graduate school, when many graduate students in subfields outside of American politics tend to be in the field. Perhaps changing the timing of POLMETH would encourage a broader swath of students in all subfields.

I am not advocating that we lower the methodology bar in order to include a broader swath of applications in political methodology. On the contrary, I am encouraging political methodologists to step up to the challenge of answering questions in political science where data is not as easy to access and where methodological questions are paramount. To signal that we are serious about questions, not just selecting on type of methods, POLMETH could also include panels that focus on methods for applications in difficult or data-scarce areas of political science. This would recognize that questions with less data often require more creative methods and we as political methodologists should also focus on answering these questions as well as those with easy data access.

To students of political science: it’s never been a better time to be a political methodologist studying questions in subfields less represented at POLMETH. There are new opportunities to develop and understand methods that travel around the world. If you can, join us at the next POLMETH meeting. You will push us to think more broadly. We hope that we can be helpful in transferring tools across subfields and disciplines that will be helpful in your research.