Political scientists generate, analyze, and deploy myriad types of qualitative data to support their claims and conclusions. To produce those data, they conduct interviews and focus groups, collect archival documents, download video clips, record music, take photographs of varied phenomena, and use many other techniques. The heterogeneity of the resulting data, and the different ways in which authors deploy them to support the arguments in their scholarship, mean that authors face challenges in making their work more transparent.
Annotation for Transparent Inquiry (ATI) is a new approach to transparency in qualitative and multi-method research that can help authors to address those challenges. ATI empowers authors to demonstrate the rigor of their work and to make it more comprehensible and evaluable, democratizing access to and fostering the accumulation of knowledge.
ATI empowers authors to demonstrate the rigor of their work and to make it more comprehensible and evaluable, democratizing access to and fostering the accumulation of knowledge.
This symposium is composed of brief articles written by some of the scholars who have pioneered the use of ATI in political science.Footnote 1 As these contributions demonstrate, ATI facilitates transparency in multiple types of qualitative inquiry as well as in multi-method scholarship. Authors discuss how ATI strengthened their work, consider the difficulties and costs that using ATI created, and propose best practices for using ATI. They also highlight new intellectual frontiers that ATI can help scholars to cross with regard to both producing and consuming knowledge. Taken together, the contributions suggest that ATI holds the potential to considerably increase the transparency, as well as the rigor and richness, of qualitative scholarship. Although all contributors acknowledge that using ATI requires time and effort, they argue that this effort is manageable and worthwhile—especially if scholars anticipate using ATI and plan accordingly.
RESEARCH TRANSPARENCY AND ATI
Research transparency—that is, describing in detail how the data underlying a piece of scholarship were generated and analyzed and sharing those data ethically and legally—is an important goal across scientific disciplines. Broad acceptance of the value of transparency springs from four key contributions that it makes to the reliable production of new knowledge. First, open data and materials allow scholars to illustrate the power and rigor of their work. Second, transparency increases the clarity of scholarship, augmenting the comprehension of diverse readers, from scholars to policy makers to activists and other key actors. Third and relatedly, transparency facilitates evaluation, allowing scholars to assess the limitations and recognize the strengths of one another’s research, potentially enhancing their confidence in its claims and conclusions. Fourth, and most generally, transparency democratizes access to knowledge and promotes its accumulation, encouraging and empowering scholars to build on one another’s findings and to reuse shared data in their research and teaching.
Recent investments in sociotechnical infrastructure—that is, capabilities, arrangements, and institutions arising from the interdependent relationship between social and technological systems (Sawyer and Jarrahi Reference Sawyer, Jarrahi, Topi and Tucker2014)—have empowered unprecedented levels of transparency and addressed challenges in achieving it (Elman, Kapiszewski, and Lupia Reference Elman, Kapiszewski and Lupia2018; Miguel et al. Reference Miguel, Camerer, Katherine Casey, Esterling, Gerber and Glennerster2014). In the social sciences, most of this new capacity has been directed to and mainly facilitates transparency in quantitative inquiry. Due to the varied forms that qualitative data take (i.e., text, images, audio, and video) and the way they are deployed in published work, scholars whose research entails generating and analyzing such data require different mechanisms and strategies to achieve transparency (Karcher, Kirilova, and Weber Reference Karcher, Kirilova and Weber2016; Mannheimer et al. Reference Mannheimer, Pienta, Kirilova, Elman and Wutich2019).Footnote 2
Qualitative data typically are analyzed, and used to support claims in scholarship, individually or in small groups: an archival document, interview recording, newspaper article, video clip, or small groups of such sources serve as distinct inputs to an analysis. Yet, increasingly stringent journal-article word-count limits and publisher concerns about the length of book manuscripts make it difficult for authors to use large amounts of text as supporting material. This complicates the deployment of the very forms of evidence that are the lifeblood of much qualitative work. Furthermore, rather than presenting results in a single matrix as authors of quantitative work do, those who deploy qualitative data often interweave data, analysis, and conclusions across the span of their written work. Figure 1 compares how data are analyzed and deployed in quantitative versus qualitative research.
Figure 1 Data and Analysis in Quantitative versus Qualitative Research
Note: Quantitative analysis from Bolsen, Leeper, and Shapiro (Reference Bolsen, Leeper and Shapiro2014); qualitative analysis from Snyder (Reference Snyder2015).
ATI accommodates these features of qualitative inquiry. Developed by the Qualitative Data Repository (QDR) and the software nonprofit Hypothesis, ATI is a flexible, versatile approach to achieving transparency in qualitative and multi-method research.Footnote 3 ATI uses open-web annotation, which allows for the generation, sharing, and discovery of digital annotations across the web. Authors who use ATI digitally annotate particular claims or conclusions in their manuscript. Annotations may include full citations to the data sources underlying empirical assertions; “analytic notes” that clarify how authors generated or analyzed their data and/or how they support their inferences or interpretations; excerpts from data sources; and, potentially, links to the data sources themselves (figure 2). For scholars who use lengthy footnotes to clarify and illustrate points, ATI annotations are a natural extension of their writing practices (see also Gerring in the symposium conclusion).
Figure 2 ATI at a Glance: How a Passage in the Text of an Article Annotated with ATI Appears to Readers
Source: O’Mahoney (Reference O’Mahoney2019).
Annotations are displayed on the same publisher web page (or PDF) as the text of the digitally published article or book that they accompany. They can be used by any publisher on any publishing platform. Annotations (and underlying data sources) are curated and preserved by a repository where readers can further investigate and potentially access them for reuse (e.g., for evaluating the relevant publication, answering new questions, or teaching). Immediately proximate to the text and instantly available to readers, annotations enhance scholarship in many ways, all without increasing word count.
As the contributions to this symposium show, ATI also can be used to increase the transparency of multiple types of qualitative and multi-method research. ATI annotations can help authors to show the data that underlie their claims or to provide details about how they collected or analyzed the data (as noted previously); to discuss reflexivity and positionality; or to provide additional context for a case study. Thus, ATI is a versatile approach to making qualitative research transparent (developing a methodological appendix is another approach). Of course, as some symposium contributors note, scholarly use of ATI can have positive consequences that extend beyond transparency. Moreover, the technology used by ATI—open-web annotations—has broader applications: it can be used to comment on or carry out peer review of digital manuscripts, and it can facilitate teaching and learning. Nonetheless, the core goal of using ATI is to enhance the transparency of qualitative and multi-method work.
ATI’S POTENTIAL
As highlighted in the contributions to this symposium, ATI has several distinctive advantages as an approach to transparency, and it produces various broader benefits for research. Myrick suggests that anticipating the use of ATI encouraged her and her coauthor to approach the collection and interpretation of evidence more methodically than they otherwise might have done. Also, as Mayka and Myrick both note, ATI helps authors to “invite readers in” to the social world of study by enabling authors to provide rich description and use participants’ own words as evidence. Simultaneously, ATI enables scholars to ethically pursue openness by helping them to protect their research participants (because the data underlying ATI annotations are stored in a trusted repository).
Symposium contributors also found that ATI increases research integrity: by helping scholars to identify and encouraging them to resolve contradictions between the argument they are developing and their evidence—that is, by holding them accountable to their data—ATI helps them to overcome “motivated reasoning,” thereby increasing the accuracy of their assertions and inferences (Mayka, Milonopoulos, and Siewert). Moreover, contributors suggest that ATI increases the accessibility of their scholarship: by acting as a digital exoskeleton, annotations can provide technical details of the method used in a manuscript, increasing the work’s evaluability and its utility to readers interested in learning how the relevant method works. Furthermore, because of the proximity of annotations to the text to which they relate, they are easier to access than are alternative transparency mechanisms such as footnotes and methodological appendices, making viewing annotations less disruptive to the reading experience. Indeed, readers who are mainly curious about a work’s core findings and insights can hide the annotations completely (Milonopoulos, Myrick, and Siewert). In aggregate, the symposium contributions demonstrate that ATI is an adaptable, flexible approach to transparency that can be useful in many types of qualitative inquiry.
ATI helps [authors] to overcome “motivated reasoning,” thereby increasing the accuracy of their assertions and inferences.
Of course, given ATI’s novelty, challenges remain. Various contributors (e.g., Mayka and Siewert) highlight the time that they dedicated to creating annotations and thus did not dedicate to other career-enhancing activities. There also is consensus on the importance of establishing a solid workflow to facilitate ATI’s integration into research processes (Mayka, Myrick, and Siewert).Footnote 4 Relatedly, some contributors (e.g., Mayka) struggled to identify the best point in the writing process at which to annotate, noting how early annotation can be helpful but also distracting and potentially inefficient because some initial annotations may be cut during the editing and review process. Likewise, contributors grappled with what to annotate—and how to determine if they are annotating “too little” or “too much,” given the absence of consensual norms regarding the use of ATI (Milonopoulos). Siewert offers a promising set of criteria that authors can use to evaluate and calibrate their use of ATI. Contributors also note how uncertainty about whether and how journals will integrate annotations into the peer-review and publication process complicates the composition of annotations (Myrick). We agree that many key questions about ATI and its use remain unanswered. As the approach becomes more widely adopted, norms for its effective use in multiple types of qualitative and multi-method inquiry will emerge. We hope this symposium represents a step toward that goal.
CONCLUSION
Qualitative research makes significant contributions to political science. Nonetheless, scholars who conduct qualitative inquiry sometimes have difficulty convincing other researchers of the rigor and relevance of their work. In part, this challenge arises because the meticulous research procedures and practices that scholars of qualitative work use to generate and analyze their data sometimes remain invisible, which complicates the assessment of the evidentiary value of the data and the quality of the analysis. At the core of these difficulties is a disconnect between the way in which scholars who conduct qualitative research build arguments and deploy evidence, on the one hand, and the way in which social science scholarship is represented in articles and books on the other.
ATI provides scholars who conduct qualitative inquiry with a way to reveal the power of their research, make it more understandable, and facilitate its careful evaluation. ATI does so by offering authors a platform on which to discuss in detail how they generated and analyzed their rich data; to demonstrate how those data support their claims; to describe the evidentiary value of their data; and to make those data available (when this can be done ethically and legally). In summary, ATI opens a window on the inferential soul of qualitative inquiry.
By inviting and broadening access to key insights about qualitative methods, to the knowledge that qualitative and multi-method scholarship produces, and to qualitative data, ATI encourages intellectual democratization and addresses inequities in data access. Furthermore, ATI offers a fresh approach to transparency and a new way of thinking about its goals. Because renewed emphasis on transparency in the last few decades was catalyzed by and remains linked to the inability to replicate or reproduce empirical scholarship (i.e., the “replication crisis”), a central goal of many transparency initiatives has been to identify bad actors. ATI, by contrast, lifts up good actors. Rather than facilitating the identification of error in completed scholarship, ATI encourages ethical research practices by helping scholars to make their work more transparent as they produce it. This normative orientation sends a subtle but powerful signal and—in combination with the data protection, preservation, and promotion that ATI encourages and facilitates—makes ATI an approach to transparency around which scholars from diverse perspectives can rally. As such, ATI holds the potential to bridge the divides over transparency that have developed among scholars who generate and analyze qualitative data, thereby contributing to the health of contemporary social science.
Rather than facilitating the identification of error in completed scholarship, ATI encourages ethical research practices by helping scholars to make their work more transparent as they produce it.
ACKNOWLEDGMENTS
We are extremely grateful to the two scholars who reviewed this article and the other contributions to the symposium for their incisive and productive comments, which made our contribution and the symposium as a whole much stronger. We also thank the symposium contributors whose work inspired ours. This article is based on work supported by the National Science Foundation under Grant Nos. 1823950 and 1946272.
