Technology-Enabled Social Scientific Collaboratories

When can evolving communication technologies fuel successful large-scale scientific collaboration? A common framework for thinking about such matters comes from Mancur Olson's (Reference Olson1965), Logic of Collective Action. It begins by documenting a familiar conflict between individual desires and group goals: “though all of the members of the group … have a common interest in obtaining this collective benefit, they have no common interest in paying the cost of providing that collective good” (21).

Olson outlines three conditions under which self-interested individuals will work together to pursue shared interests: (1) when the group size is small, (2) when some form of coercion may be put into use, or (3) when there are special incentives for collaboration. This conclusion depends on a key assumption: the larger the size of the group, the more costly the organization and oversight of the group. Underlying this assumption is the premise that if large numbers of people are going to work together for a common cause, they may be geographically dispersed—and such dispersal will increase the marginal cost of communicating with, or monitoring, additional members.

Evolving communication technologies permit changes to such assumptions in scholarly communities because they reduce many costs that are pertinent to organization and oversight (Lupia and Sin Reference Lupia and Sin2003). For example, scientific organizations, such as APSA, the Midwest Political Science Association, the American Association for the Advancement of Science, and the Consortium of Social Science Associations (COSSA), use electronic communications to motivate potentially interested scholars to contribute to online forums and to notify individuals of public gatherings. At minimal cost, such groups now regularly transmit messages that can be viewed instantly by massive audiences. For most of the twentieth century, transmissions of such scale could be made only by a few world leaders, public figures, and media moguls—and even for these people such opportunities were available only at select moments and were expensive. Now, the opportunity to send messages that can, in principle, be viewed by worldwide audiences is available to tens of millions of people at very low costs.

When evolving technologies reduce organizational costs in this way, formerly impossible large-scale collaborations become possible. The possibility is important because, almost universally, scientific communities are dispersed over great distances. It is rare that all scholars working on a particular topic are at the same university or even in the same country. New technologies reduce many barriers to broad and vibrant scholarly communication.

The ANES and TESS show how integrating evolving communication technologies into collaborative strategies can provide new research opportunities to scientific communities. While the two projects are distinct, they have four common attributes of interest.

1. 1. They are large scale—their respective inputs and outputs are due to interactions amongst hundreds of scholars from many geographically dispersed locations.

2. 2. They are funded by the U.S. National Science Foundation, whose ambitions for these projects are quite rigorous.

3. 3. The projects' recent successes are due, in large part, to the use of communication technology to create (in the TESS case) and expand (in the ANES case) the range of scholarly interactions.

4. 4. Both projects originated in political science.

This origin is relevant because political science is a field in which scholars from different disciplinary viewpoints regularly interact. Like much work in political science, TESS and the new ANES build from a deep faith in the potential power of interdisciplinary approaches and a belief that new technologies are part of tapping such potential more effectively.

TESS

Many social science experiments are run in university-based laboratories. Their subjects are often undergraduates or members of the communities where research universities are located (i.e., convenience samples). Such samples leave some researchers vulnerable to claims that their results do not generalize to broader and more diverse populations (see, e.g., Sears Reference Sears1986).

TESS is an NSF-sponsored multidisciplinary infrastructure project that allows individual investigators or teams of researchers to embed experiments in nationally representative telephone and Internet surveys. It provides a new way for researchers to conduct experiments without the restrictions of convenience samples (Brady Reference Brady2000, 53). It builds from a collaborating research model developed by Paul Sniderman and others in which multiple investigators place distinct experiments on a single survey. This strategy leverages scale economies in a way that dramatically reduces the costs of producing these innovative forms of research.

TESS's primary goal is to allow investigators to conduct rigorous experiments on large, nationally representative populations. Such opportunities are particularly valuable to scholars who want to make strong claims about causality, but lack the resources or expertise to field a large experiment on samples other than those of convenience. TESS gives scholars with new and innovative ideas the means to conduct path-breaking experiments.

A key attribute of TESS is its openness: any graduate student or faculty member can submit a proposal. All proposals to TESS are peer reviewed through a double-blind process, and proposal authors receive written feedback. TESS then implements the proposals it accepts. It takes responsibility for programming, pretesting, and conducting the experiment for the investigator. All of this is done at no charge to the proposal's authors. Upon completion of the data collection, the proposal's authors receive exclusive access to the data for nine months. Thereafter, their proposal, their data, and a brief description of their results are made available on the project's web site, http://www.experimentcentral.org.

By several measures, TESS has been quite successful. By its third year of data collection, the project was receiving over 100 proposals annually. As of this writing, it has conducted experiments for over 200 researchers from a wide range of scientific fields—spanning not only the social sciences but also fields like law and public health (see, e.g., Davis and Fant Reference Davis and Fant2005). Indeed, TESS has allowed innovative researchers to evaluate many different causal hypotheses. TESS experiments have shown effects by randomly varying across experimental groups: the information presented to subjects in vignettes, question wording, images, and monetary incentives. Researchers have also used TESS to integrate video and pictures into experiments conducted on large, geographically distributed subject pools.

In addition to the opportunities described above, TESS has produced several notable byproducts. One is a new archive of experimental research. This archive, which is part of the project's web site, contains all TESS experiments for which the nine-month exclusivity window has expired. Here, researchers and students can make use of the research designs and information described above to run new analyses, replicate original analyses, or conduct mode comparisons. There are many opportunities for scholars to use existing TESS experimental designs to explore a wider range of questions than the experiment's designer originally conceived.

Another byproduct is that the TESS proposal review process has become a virtual mentoring network. In the review process, the most frequent occurrence is the proposal of a young, ambitious, but largely inexperienced scholar being reviewed by more experienced scholars in his or her field. In this context, TESS reviewers offer frank and constructive advice. Such interactions can be of special value to young scholars. To see how, consider the fact that some junior faculty and graduate students are fortunate enough to be in settings where they get good advice about research design. But many others do not—and even those who do get good advice from a senior scholar in their department may get even better advice from someone far away. Moreover, the advice that TESS reviewers give comes at an early stage of the process—where important design decisions can be made. This is unlike the review processes of most scholarly journals, where key research decisions were made long ago and cannot easily be adjusted—if they can be changed at all (e.g., if you've run a survey of the 2008 election, then you cannot use advice received in a later year about how the survey should have been designed to save the original study). TESS, in effect, has created a new virtual mentoring community in which geographic distance is less of a barrier in young scholars' attempts to learn from others' experiences.

With such achievements in mind, many people are surprised to learn that TESS is “a virtual corporation.” For most of the project's history, TESS has not had even a single dedicated office. Indeed, TESS would be a paperless operation if it were not for the fact that human subject committee approvals and contracts with survey firms must be registered on paper. But this paperwork requires only part of an average sized file-cabinet. Otherwise, all aspects of the project are stored, monitored, and executed via a suite of web-based utilities that were developed explicitly for the project by Pawandeep Lamba and the Ohio State University. TESS's principal investigators and staff can oversee the project from anywhere that they can access the Internet. TESS staff can view every proposal that TESS has ever received, they can read every peer review that has been submitted, they can track the timing of critical project activities, and they can access many previous project-related correspondences. By leveraging the Internet in this way, TESS has run with a small staff of one to three part-time employees and it has been able to devote most of its resources to data collection rather than administration. TESS is an example of a case where a single NSF grant combined with the use of evolving communication technologies generated hundreds of new opportunities for exploration, mentoring, and collaboration amongst geographically dispersed scholars from across the social sciences.

ANES and the Online Commons

The ANES is a project that is now nearly six decades old. Its origins trace back to the late 1940s when University of Michigan researchers created interview-based election data capable of benefiting a wide range of research hypotheses. Today, ANES is widely used and many people regard it as the “gold standard” of scientific election studies. Thousands of scholarly articles and books use ANES data and findings and many aspects of its design have been used to improve election studies around the world. For more on the project, see http://www.electionstudies.org.

This success, however, generated expectations of the project that were not being met. The centrality of the ANES data collection amongst researchers, students, journalists, and members of the general public who are interested in high-quality survey-based explanations of election and choice related phenomena generated a huge demand to have questions of particular interest placed on the surveys. At the same time, ANES's means for selecting questions for inclusion on its survey interviews were not transparent to many users. The lack of transparency invited suspicion about the project's openness to new ideas. While the project endeavored to incorporate a wide range of ideas into ANES questionnaires, the means for communicating and collaborating with other scholars were limited.

In 2004, the National Science Foundation challenged scholars to update the ANES in ways that would increase its value to scholars. Part of the proposal put forward by Jon Krosnick and Arthur Lupia entailed leveraging evolving communication technologies to create a new large-scale collaboratory, the ANES Online Commons, that would be used in the development of ANES surveys. The NSF accepted this proposal and ANES has since implemented it.

The Online Commons is a key part of a strategy to improve the breadth and quality of scientific input that goes into ANES survey development, as well as the public accountability of ANES decision makers. The idea is based on the “open source” method of technological development. As Neteler and Mitasova (Reference Neteler and Mitasova2004) describe:

ANES's strategy is to create a forum that would allow similar dynamics to improve the content and value of election-related surveys. To the best of our knowledge, this kind of technology-driven, large-scale collaboration for constructing an academic survey has never been attempted before.

Individuals or teams of researchers are able to submit questions for placement on the survey, to comment on the proposals that have been submitted, and to express their level of support for the inclusion of specific proposals on the ANES survey instrument. All proposals that are sent to the Online Commons, and all comments about such proposals that are posted to Online Commons message boards, are available for everyone to see. The Commons thus allows for the improvement of the ANES survey instrument and it allows individuals to obtain feedback from a broad community of scholars on questions they may include on their own survey instruments.

The Online Commons is structured to encourage participation from this broad community of scholars. To make a proposal or post a comment on the Online Commons, one must become a member. Any individual faculty member, student, survey researcher, or social science professional can apply for membership into the Commons by submitting a short application and by agreeing to the Code of Conduct. Community members can then select up to two user names: one that is publicly identifiable, the other that allows anonymity. While community members' verifiable information—full name, e-mail address, phone number, and organizational affiliation—is not publicized within the community, administrators keep it on file. This membership structure enables members to participate honestly within the norms of a deliberative community.

For each ANES survey, the Online Commons is open for a discrete period of time. After a period ends, the principal investigators and Board of Overseers review the proposals. In addition to comments received from other scholars on the Online Commons, each proposal author receives individualized written feedback about his or her proposal from the principal investigators (a process that itself is made feasible by the use of databases where comments about each proposal can be organized for quick and effective retrieval).

The Online Commons is designed to produce (1) decisions about study content whose logic is transparent and public, and (2) substantial public debate and input about conceptualization, theory, and measurement before each study is fielded. This process constitutes a stark contrast with existing practice in the development of many surveys, in which conversations about questionnaire construction have been more sporadic, occur mostly after the data are released, and are conveyed primarily in ways that make it difficult or impossible for the survey producers to react and adapt in a timely manner. The Online Commons is designed to increase participation in, and the transparency and accountability of, the ANES decision-making process.

At the same time, drastically lowering participatory barriers can have adverse consequences for large-scale collaborations. If, for example, individuals are able to submit very lengthy proposals, then the likelihood of others reading proposals in their entirety decreases. If prospective visitors were to anticipate many long treatises, participation in the community would become less beneficial. It is therefore important to structure the community to impose rules for a submission to encourage responsible use of the resource. The Online Commons enforces this structure by imposing timeframes community members can participate within and length restrictions (10 pages) for proposals.

From 2005–2008, the ANES used its Online Commons to solicit proposals for questionnaire content four times. The first use occurred from March to June of 2006. The most recent uses concluded in February 2008. Over 600 scholars from dozens of scholarly disciplines participated in the Online Commons. Many contributors were graduate students and junior faculty. Altogether, the process generated proposals for over 3,000 questions. The breadth and depth of these ideas generated ANES surveys with many dynamic new elements.

Like TESS, the ANES Online Commons is a paperless operation. It was developed by the ANES staff at Stanford and Michigan under the leadership of David A. Howell. The Online Commons integrates publicly accessible message board templates with utilities and security features that protect participants and allow ANES staff to effectively manage the many submissions. The Online Commons represents the latest step in an evolving relationship between ANES and the broader community of scholars. It leverages new communication technologies in ways that allow more expansive input into survey development than ever before.

Challenges for Future Endeavors

People who seek to use the Internet to harness the potential power of large-scale, geographically-distributed interactions must keep the competing demands for prospective participants' attention in mind. TESS and ANES attempt to manage timing challenges through the strategic use of deadlines.

In the early months of TESS, for example, many scholars indicated an interest in the project and expressed a desire to send a proposal, but few actually did. The TESS principal investigators wondered whether its rolling acceptance policy was contributing to the delays. Soon thereafter, they announced a “Special Competition.” The requirements for the competition were almost identical to their normal requirements. The main difference was that the competition publicized a deadline.

In its first three years, TESS ran three “Special Competitions.” In every case, the number of proposals received in the days leading up to the deadline surpassed the number of proposals it received during the rest of the year. TESS subsequently adjusted many of its procedures to work from a new premise about individual participation—researchers pay more attention to requests for participation when the requests are accompanied by a firmly stated deadline. Thereafter, TESS used deadlines strategically to generate interest in the project and to give greater predictability to variations in its workload.

The ANES has had a similar experience. ANES received over 75% of the proposals within a week of the deadline during each of the first two runs of the Online Commons. Roughly 50% arrived within 48 hours of the final deadline.

In addition, while over 600 people registered for the Online Commons, and many more viewed the proposals, no proposal had more than 11 comments and many proposals received no comments at all. Moreover, conversations around the proposals were generally supportive in nature. The ANES principal investigators anticipated more debate.

During the final days of the first Online Commons, when the magnitude of the deadline-oriented proposal submission flow was becoming apparent, ANES extended the deadline for comments on proposals by one additional week. While this move generated some additional comments, the magnitude was less than the project had hoped for. From that experience, ANES moved in its subsequent versions of the Online Commons to announce separate deadlines for proposals and comments, with the comments deadline following the proposal deadline by two weeks. While this strategy increased commentary by some amount, it is still the case that the scholarly community devoted far more effort to developing their own proposals than to evaluating the proposals of others.

At the present time, we believe that ANES users would benefit from greater participation in open discussions about the value of various questions. But this is one area where the magnitude of participation is less than what we had hoped for. It may be that professional incentives and social norms within the social sciences will prevent the full benefits of this particular attempt to encourage Open Source development from being realized. This is a problem of collective action that needs further research and experimentation.