The goals of transparency and replicability resonate strongly among most scholars who conduct quantitative work in the social sciences. For this community, the ideals seem both important and realizable. By contrast, the community of scholars who conduct qualitative research has approached these ideals with some apprehension, voicing concerns about feasibility as well as desirability.
The advent of Annotation for Transparent Inquiry (ATI) probably will not change anyone’s view about the desirability of transparency and replicability. However, it may change their view about feasibility.
The advent of Annotation for Transparent Inquiry (ATI) probably will not change anyone’s view about the desirability of transparency and replicability. However, it may change their view about feasibility.
The contributors to this symposium offer a range of reflections on ATI—mostly positive and mostly based on personal experience. These are the pioneers, and pioneers are not like the rest of us, so we must be wary of drawing conclusions on an obviously biased sample. Nevertheless, I am convinced that ATI is a practical approach that we all can learn and that—if widely used—is likely to advance the goal of scientific cumulation.
In this concluding article, I emphasize ways in which ATI continues and extends traditional practices of qualitative research. Although the approach is novel, the goals it achieves are not. Clarification and commentary on sources are intrinsic components of qualitative research. From this perspective, ATI is not a break from tradition but rather an extension of a venerable tradition. Acknowledging this may ease the path of acceptance for those who are suspicious of flashy, hi-tech innovations. It also should deflate some of the unrealistic expectations that seem to accompany any new technique. ATI will not deliver us from the difficulty of interpreting complex and ambiguous sources; however, it will make those interpretations clearer, easier to follow, and easier to dispute. In the long term, this should lead to greater scholarly consensus and greater credibility.
HOW NEW?
It is easy to become beguiled by the nifty technology associated with ATI. However, we must not lose sight of the fact that this technology is a tool for something that qualitative scholars have striven to achieve: establishing the validity of their data and allowing others to follow their steps—the core ideas behind transparency and replication.
Traditionally, this task is handled in footnotes, appendices, annotated bibliographies, and sections of a manuscript (e.g., a chapter, a prologue, or the acknowledgments) that comment on sources and how they support (or conflict with) the author’s claims. From this perspective, ATI is simply an extension of current practices.
If ATI is novel, it is by degrees. It offers more space for commentary (which, for all intents and purposes, is limitless) and the opportunity of linking directly to sources, which may be of many varieties (e.g., textual, oral, pictorial, or video)—a point explored by Siewert (Reference Siewert2021). These links flow directly from the text, only a click away, so there is no need to search online or depend on libraries and archives. The links can be accessed directly and instantly. (Ultimately, we can hope that a link will take us to the precise location—of what might be a long manuscript or transcript—that is relevant to the writer’s argument.)
This is no mean feat. It has been said that qualitative research suffers a reputation problem within the social science community. “You can prove anything with a case study” is the view of many (Moravcsik Reference Moravcsik2010, 29). The problem is that readers are at pains to evaluate the evidence provided by the author. Indeed, without direct access to the evidence, readers are at the mercy of the author, whose interpretation is unchallenged and unchallengeable (at least, without significant footwork). Insofar as ATI facilitates access to evidence, it makes the ideal of transparency manifest and the ideal of replication possible. In this respect, ATI seems well suited to achieve what every qualitative scholar aims to achieve. If scholars are confident of their interpretation of sources, they should welcome any approach that facilitates easy access to those sources for readers so that their interpretation can be vetted—and, it is hoped, consensus can be reached within the scholarly community.
That said, ATI should not be oversold. It is no panacea and does not solve—or even attempt to solve—many of the challenges currently facing qualitative research. This includes ascertaining the validity of sources, knowing how much source evidence should be provided, knowing how to deal with divergent sources, and figuring out how to secure the confidentiality of sources. It is important to understand these limitations lest the advantages of ATI be overhyped—the dismal fate of many new techniques. These challenges are orthogonal to the goals and functioning of ATI.
WHAT DOES DATA VALIDITY MEAN?
In quantitative research, data validity is fairly easy to achieve (at least compared to qualitative research). We cite the original source (if an extant dataset) or state the protocol by which data were collected (if an original dataset). There is no attempt to deal with the validity of the data at the level of individual observations. For example, the Varieties of Democracy (V-Dem) project describes in great detail how expert coders are recruited; how they perform their coding tasks; the questionnaire they fill out; and the method of aggregating datapoints across coders, years, and countries (Coppedge et al. Reference Coppedge, Gerring, Glynn, Knutsen, Lindberg, Pemstein, Seim, Skaaning and Teorell2020). However, there is no attempt to ascertain whether an individual datapoint (i.e., the coder’s rating or all coders’ ratings for a particular variable/country/year) is correct. Claims to validity rest primarily on following a defensible procedure, not on the results obtained by that procedure.
Not so for qualitative researchers, who must defend each datapoint. In many respects, this is a higher standard than that which quantitative research faces. At any rate, it is a more ambiguous standard.
The saving grace is that, typically, there are not as many cases to keep track of in qualitative research. Whereas V-Dem enlists thousands of coders, millions of codings, hundreds of variables, and tens of thousands of variable/country/year estimates, a typical qualitative research project has a more narrow focus—for example, on a single or several units (e.g., countries, organizations, or groups), a single time period, or even a single event. Of course, there (typically) are numerous datapoints emanating from every case, and these datapoints are of different types. They are all (by presumption) relevant to the central inference, but they are non-comparable, which means they cannot be reduced to the two-dimensional space of a flat dataset (Gerring and Thomas Reference Gerring, Thomas, Badie, Berg-Schlosser and Morlino2011).
Establishing the validity of a single case is quite different from—and, arguably, more challenging than—establishing the validity of a dataset. For the latter, we typically explain the method of data collection, examine possible sources of bias, report reliability tests, and perhaps run convergent validity tests. For the former, there are no well-established rules.
Complicating matters further, in qualitative research, the line between data collection and data analysis usually is more difficult to define. One is entangled in the other; this means that qualitative scholars must wage a simultaneous, two-front war.
ATI will not solve any of these time-honored problems of qualitative research. However, it may nudge authors toward greater self-consciousness and integrity, a point acknowledged by authors in this symposium (Myrick Reference Myrick2021; Siewert Reference Siewert2021). “Knowing that ATI would make my underlying evidence readily accessible to readers encouraged me to consider my evidence more judiciously, and make my inferences more explicit, than I might have done in ATI’s absence” (Milonopoulos Reference Milonopoulos2021).
HOW MUCH IS ENOUGH?
Qualitative researchers seeking to explain and justify their data face the problem of deciding how much information about sources should they provide to readers. By opening the gates to endless pieces of evidence, as well as commentary on that evidence, ATI seems to invite evidence overload. (This point is raised by several symposium contributors.)
However, we cannot blame ATI for a problem that is inherent in all qualitative research. ATI is the messenger, not the cause, of this conundrum. It forces us to recognize a problem that is nascent in any venture aiming at the truth. How do we know we got it right? What do/should readers know to fairly evaluate our conclusions?
In the pre-ATI world, there were severe space limitations. After all, there is only so much information that we can fit into a footnote or an appendix—and appendices easily become unwieldy. (How do we identify the piece of a 200-page appendix that is relevant for a particular point?) Traditional referencing also is limited insofar as it does not integrate alternate media or sources as they appear in primary documentation. It is this type of limitation that makes a journal article an unfriendly location for qualitative work, which often requires extensive discussion of sources (Mayka Reference Mayka2021).
With ATI, these limitations are overcome, which means that decisions about what and how much to present are now unavoidable. Authors can no longer ignore these core questions because of space limitations. Instead, they are forced to justify their decisions. This means that each field must develop standards of evidence that are relevant to their chosen topics.
In this respect, ATI may be viewed as liberating, lifting technological restrictions that heretofore constrained scholarly communication. ATI cannot answer these questions; it only can offer an approach with which they might be answered—but this is a major step in the right direction.
The invention of the internet raised the question of what to do with essentially limitless communication options. It did not tell us whom to communicate with or about what. The same might be said (in a less revolutionary fashion) for ATI.
HOW TO DEAL WITH EVIDENCE DIVERSITY?
Another issue that ATI does not address is contrary evidence. Commonly, writers cite evidence for their proposition, leaving aside—or noting only briefly—evidence that does not support or that perhaps even contradicts their argument. This is acceptable if it is understood that evidence is being used illustratively; that is, “There is some evidence to back me up; someone out there agrees with me.” However, the claim often is stronger: “The evidence, on balance (taken as a whole), agrees with me.”
To my knowledge, qualitative scholars have not) developed a protocol for citing an array of evidence—supportive opposing, or perhaps neutral in import. It is not clear how we could or should represent this messy reality.
Naturally, if all sources were equal in status, we simply could classify them in bins: (1) supporting, (2) opposing, or (3) ambivalent. However, this is rarely the case. Typically, some sources are more trustworthy or more relevant to the question than others. Moreover, some datapoints probably should be regarded as redundant because they rely on the same primary source or the same informant. This type of judgment requires interpretation and, hence, commentary, which is when ATI is helpful (Mayka Reference Mayka2021).
The technology alone will not solve the problem of divergent evidence; it merely offers space for solving the problem. Researchers will need to develop a protocol for how to address the ubiquitous situation of conflicting pieces of evidence. However, at least now there is a technology that makes it possible to have this discussion in the depth and detail that it deserves, which means there is no excuse (except for time burdens on researchers and readers) for not doing so. It also means that qualitative researchers can present a more nuanced and faithful view of the evidence. They do not need to overclaim because they can explore the subtleties behind their conclusions. This approach should prove more persuasive than the claim that is implicit in much qualitative research: “Trust me, I have pored over all the relevant evidence and explored all the angles.”
The technology merely offers space for solving the problem. Researchers will need to develop a protocol for how to address the ubiquitous situation of conflicting pieces of evidence.
HOW SHOULD CONFIDENTIALITY BE ASSURED?
All scholarship, and especially qualitative scholarship, faces a problem with respect to the confidentiality of sources. If informants must speak openly (i.e., “on the record”), they may speak untruthfully or may not speak at all. Yet, if informants are granted confidentiality, it is not possible to replicate the findings (at least not in an obvious sense). Every scholar who has conducted research with human participants has faced this dilemma, an issue discussed by Myrick (Reference Myrick2021).
It goes without saying that ATI will not solve this problem; it does not make it better or worse. However, it does make it more real. In the pre-ATI and pre–Qualitative Data Repository (QDR) world, it was easy to avoid the issue because publication venues did not have established protocols for making qualitative research materials public. There was no place to put our interview transcripts, field notes, and archival documents. As a result, informants often enjoyed confidentiality.
Now, as a result of ATI and QDR, it is possible to post qualitative source material online so that it can be accessed by readers and to connect specific points in a published article to specific source materials. From this perspective, we might say that ATI and QDR have created a problem where none existed.
However, it is a good problem to have. Frequently, there are no problems of confidentiality that would prevent posting qualitative data so that other researchers can access them. Moreover, confidentiality sometimes can be secured by anonymizing materials. In these instances, ATI offers a useful option; where it does not, the author must demur.
The tradeoff between confidentiality and transparency is an enduring one. It is deep, it is philosophical, and there are no easy solutions. ATI forces us to recognize this, and it forces the argument out of the realm of technology and into the realm of science and ethics, where it belongs.
CONCLUSIONS
ATI offers a new approach that is designed to enhance an old set of practices—referencing and commenting on sources (i.e., how they were generated and analyzed and how they support claims). This article emphasizes the traditional nature of ATI, which represents a continuation, rather than a break, from the status quo. It follows that ATI does not solve most of the conundrums that bedevil qualitative research. It does not instruct us in how to establish the validity of our data, which pieces of information should be included and which excluded as we write up our research findings, how to address a diversity of evidence (some of which supports a thesis and others that do not), or how to protect the confidentiality of sources.
What it does—and all it does—is offer a new platform for referencing sources and commenting on those sources. The platform is useful because it is limitless in size and scope (i.e., handling different types of data). It does not interfere with the flow of the prose; readers can choose to follow the ATI trail or not. Neither does it consume scarce space in a journal article. Furthermore, the information and data that ATI incorporates are easily accessible at the click of a mouse.
However, technological innovation may pave the way for innovation on other fronts. By offering a technology with limitless capacity for posting materials and commentary, ATI expands opportunities to reference sources and comments on textual interpretation. This means that decisions about what to make transparent and what to keep hidden can be made on the merits of the case rather than on the available technology or space limitations imposed by a publisher. It is up to researchers and to research communities to make the call. This, in turn, may stimulate further thinking on these core challenges of qualitative research illustrated herein.
Although I emphasize the limitations of ATI, we also should appreciate its potential. By expanding ease and precision in the task of referencing the data that undergird an analysis, ATI surely will improve transparency and replicability in qualitative research and also may lead to clearer standards with respect to how evidence is presented and adjudicated.
Of course, as in any new technology, it is difficult to calculate the costs and benefits until people start using it. ATI must be “scaled up” to provide a proper trial. In this vein, we must wonder: How many qualitative researchers will use ATI? How many journals will facilitate its use? Will it be possible to adopt ATI for books? (This is a key issue for qualitative researchers, who prize the space afforded by a book-length manuscript. It is encouraging to learn that Cambridge University Press currently is producing its first book with ATI.) How much additional time does ATI require, once users have mastered the technology?
Another set of questions arises with respect to ATI’s reception in the broader scholarly community. Will qualitative research that uses ATI be viewed as more reliable than research that does not? How many readers will click on the ATI links and what will they make of what they find there? Will ATI stimulate actual replications? Will ATI links become fodder for debates and further research? Will it change the consumption of qualitative research (i.e., the way it is digested by readers)? Time will tell, but time will be informative only if there is sufficient take-up. ATI must be given a fair trial, and this requires researchers to adopt it—if only on a trial basis.
Qualitative researchers, annotate! We have nothing to lose but our time. Perhaps, once the procedure becomes familiar, the additional time demands will turn out to be minimal.
