CAUSES OF A CRISIS

A common misconception is that crises of research design result from insufficient preparatory work. In practice, however, graduate students spend years developing their projects. By the time they have developed a coherent research project, survived the prospectus defense, and secured travel funding, their projects have passed several rounds of rigorous review.

In reality, all empirical research projects run some chance of running into crisis. Two fundamental and unavoidable aspects of positivist social science create this risk. First, social scientists routinely import theories, concepts, and methods developed in one context and apply them to another. We identify empirical or theoretical puzzles and generate likely answers using information gleaned from other people, places, and times. Generating and testing hypotheses is at the heart of our work. However, caution is necessary. To avoid a crisis of research design, the concepts, tools, or theory must resonate with your specific cases and reflect empirics at your specific field site. Importing a dependent variable is particularly risky. Hypotheses can be tested, modified, and retested in the field. Retooling a dependent variable, however, is likely to have major implications for other aspects of the research design.

Second, political scientists tend to study contemporary phenomena; we study moving targets. Major empirical events can upend carefully constructed research plans. Unexpected, cataclysmic political developments—such as revolutions, economic crises, and major policy changes—can happen just as a researcher is about to embark for the field. As the recent Arab Spring illustrates, a case of regime stability quickly can become one of regime breakdown.

Some projects entail higher risk. Scholars who study lesser-known topics or cases are more likely to rely on imported concepts and theories when developing their projects. Studying understudied cases is an important enterprise, and one to which many scholars are committed. However, the practice also generates higher levels of uncertainty when developing a research design. No established body of literature exists that lays out the main theoretical debates for these cases. Nor is there scholarly agreement on what constitutes an empirically meaningful development. Preliminary data may be more difficult to obtain. In addition, dissertation advisors may be general experts on the topic or region, but likely are not an expert on your specific countries. Under these circumstances, a researcher is more likely to “import” theories and concepts originally developed in other contexts.

These risks are exacerbated when a long lag exists between defending the prospectus and leaving for the field. The problem is particularly acute for graduate students who rely on external grants to fund their fieldwork. Polished, well-specified grant applications are due in the fall; awards are announced in the spring; and travel may commence in the late summer or early fall. Thus, graduate students may wait an additional year before heading into the field. This lag increases the likelihood that preliminary work can become outdated.

My project scored high on these risk factors. My dissertation focused on understudied cases. More than a year lapsed between defending the prospectus and leaving for the field. Crucially, I imported the outcome to be explained: state response to protest. I wanted to study the interaction between government, opposition, and average citizens within nondemocratic regimes. These interactions usually occur behind the scenes. Mass protests and their consequences seemed to be an observable way to study these dynamics. I knew that protests had been an important phenomenon across the post-Soviet region since perestroika (e.g., Beissinger Reference Beissinger2002; Ekiert and Kubik Reference Ekiert and Kubik1999). When I defended my prospectus in 2006, the “color revolutions” had recently toppled governments in Georgia, Ukraine, and Kyrgyzstan. In Uzbekistan, by contrast, government troops had used force to quell an uprising in Andijon, massacring hundreds of protesters. These events exposed important aspects of government-opposition dynamics in seemingly similar post-Soviet countries. I inferred that studying state response to protest would expose the underlying dynamics in my case studies as well.

WARNING SIGNS IN THE FIELD

What does a crisis of research design look like? Four warning signs follow.

First, researchers should take heed if data collected in the field contradicts their preliminary coding of the dependent variable. As previously discussed, these discrepancies could occur if a major political event has transformed the situation on the ground or if your preliminary research is outdated when you arrive at your field site. Scholars also can be misled by biased sources used during preliminary research. For scholars who do international research, the sources available in the United States rarely reflect the full range of viewpoints as those on the ground. Most user-generated online content—such as Facebook updates, YouTube videos, or blog posts—reflect the opinions of younger, urban, upper-class citizens. Official policy analyses conducted by international aid agencies or the local government frequently advance a political agenda. Easily accessible research contacts often include friends-of-friends, official spokesmen, or members of a diaspora community. These groups are unlikely to be representative of the larger population. My preliminary work was based on a protest event dataset that I constructed using news articles from Azerbaijan, Kazakhstan, and Belarus. These articles were available through the US Department of State’s FBIS reports. However, most of these articles were from opposition-affiliated newspapers, which overestimated the importance of their activities.

A second red flag involves conceptualization. Fieldwork may reveal important dimensions of the dependent variable. A project may need reworking if interview data, personal observation, or quantitative evidence suggests that the outcome does not capture the full story. One interview subject in Kazakhstan, for example, told of a major protest that was allowed to proceed. Police stood by as thousands marched through the city, but undercover security agents hid inside streets and back alleys, arresting participants on their way home. Thus, the protest appeared to be tolerated, although the participants incurred harsh administrative and coercive penalties.^{Footnote 1} My conceptualization had not accounted for this more complex state response.

Third, field research may expose unforeseen anomalies in your case selection. I faced this problem with the longitudinal aspect of my case selection. I had focused my dataset on the years 2002 to 2004, which allowed for a tidy research design. During this time, the governments in Azerbaijan, Belarus, and Kazakhstan responded quite differently to protests, although they were ruled by regimes with nearly equal levels of political openness (Marshall and Jaggers Reference Marshall and Jaggers2006) and maintained similar levels of security personnel (Institute for Strategic Studies 2004). However, my interviews in the field revealed that these years were highly unusual, especially in Azerbaijan. In the early 2000s in Azerbaijan, the state did sanction hundreds of antigovernment protests, as the news reports had suggested. However, those years corresponded to a time of political transition in Azerbaijan, as power was passed from President Heydar Aliyev to his son, Ilham. This contentious issue existed both inside and outside of government, so protests were more frequent and more tolerated. Today, Ilham Aliyev has a solid hold on power. Azerbaijan’s government has tightened legal and administrative measures against expressions of public dissent. Few political protests have occurred since 2005.

The fourth, and most serious, signal that a project needs to be reworked occurs when respondents attribute different meaning to the events under study. In both Kazakhstan and Azerbaijan, I heard during interviews that protests were not significant. When asked whether protests were important to understanding the political process, one expert responded, “In practice and concretely, in Kazakhstan they are not important. They don’t play a role in politics because no one goes to protests.”^{Footnote 2} Another person summarized that “much of politics in Kazakhstan is a performance (spektakl’)” designed to impress outsiders.^{Footnote 3} In Azerbaijan, many protests in the early 2000s were described as “small” and “not significant,” despite their heavy coverage in the media.^{Footnote 4} These comments implied that protests were not the most accurate lens to study government-opposition relations in these countries.

REDESIGN IN THE FIELD AND BEYOND

Researchers need to make a choice when confronted with these warning signs. They may proceed with the original research plan or choose to shift the design of the research project. Retooling in the field is imperative when your original project is premised on data or assumptions that were factually incorrect. But the decision is rarely so straightforward. Sometimes other mitigating factors must be considered. For example, retooling is advised if these warning signs are accompanied by a quest for data that turns out to be hidden or nonexistent (Chambers-Ju, this symposium).

Navigating this decision requires close attention to your own motivations. What drives your interest in this project? Some projects find purpose in making a theoretical contribution–—by resolving a theoretical debate, or elucidating an overlooked concept. Other projects are motivated by the potential to make an empirical contribution—shedding light on an understudied set of cases, or highlighting an underappreciated development. Indeed, successful research projects accomplish both. Nonetheless, identify the unique contribution of your research and your motivation for pursuing it.

I considered sticking with the original research design. After all, my preliminary research was not wrong; it just did not capture the whole story of politics in these countries. Doing so, however, would have left two key problems unsolved. First, given the closure of political regimes in these countries since 2004, it was difficult to collect high-quality data from my interviews. Furthermore, I could not shake the feeling that, across my cases, these individual protest events were embedded within larger patterns of government-opposition interaction. It seemed less accurate to consider a narrow time period or to focus on the micro-level analysis of protest events. With great trepidation, I abandoned my neatly controlled research design. I broadened the scope of the project in ways that I hoped would capture some important aspect of politics in these countries. Doing this required reworking the design in the field as well as continued revisions after I arrived home.

RETOOLING IN THE FIELD

Here are four steps I took while in the field.

FURTHER REWORKING AT HOME

The process of reconstructing my project continued after I returned to the States. I continued to write memos as I processed and analyzed the data. For months, I reworked the argument and revised the analytical framework before writing the first draft chapter. In the end, the overall point of inquiry remained similar from my prospectus to the dissertation, and ultimately to the book manuscript. The project continues to ask why some nondemocratic regimes give political opponents significant leeway to organize, while others enforce strict limits on these activities. However, the manuscript uses political opposition groups as the unit of analysis, rather than certain activities. I conceptualize political opposition quite broadly—as any organization that criticizes the government—a definition that reflects the situation on the ground. The causal argument is something that I would not have predicted before fieldwork. Based on data collected in the field, I argue that these policies can be traced to divergent patterns of state corruption. The Logic of Kleptocracy thus contributes to the growing literatures explaining variation among non-democratic regimes and the sustainability of non-democratic rule.