Introduction

Misconduct in scientific research is far from new. From the “discovery” of the so-called Piltdown Man in the early 20th century to the more recent case of social psychologist Diederik Stapel, stories about, and accusations of, fraudulent behavior regularly remind us that academics are no strangers to deviance. What is new, however, according to Mario Biagioli and Alexandra Lippman, is “the growing reliance on institutional metrics of evaluation,” which “creates [the] conditions of possibility” [1] for a novel type of deviance: “manipulations that are qualitatively different from traditional academic misconduct” [3]. Not “epistemic crimes” per se, such as fabrication, falsification or plagiarism, but “bureaucratic crimes” aimed at manipulating “the publishing system itself” [3]: submitting fake peer reviews, setting up citation rings among authors and editors, or else hacking a journal’s database, for instance, are distinct from epistemic claims “in the sense that they concern the publication process and the impact of the claims, rather than a manipulation of the content of the publication” [2].

Gaming the Metrics is the first book to be entirely devoted to this relatively new phenomenon. As such, it is an important contribution to the social studies of science. Its intended scope is wider, however, as it not only brings together contributions from scholars who are anchored in diverse disciplines (sociology, scientometrics, computer science, history, communication, neuroscience, law, etc.), but also from professionals gravitating around academia. Some contributors intervene to analyze and understand metrics more deeply, while others act primarily as reformers. We shall see that this strain between observation and intervention is visible in the treatment of certain topics. The 21 resulting chapters are organized in four parts.

Beyond and Before Metrics addresses the history of publication metrics––the expectations but also the fears that surrounded them early on. Alex Csiszar reminds us that Merton warned Garfield that his Science Citation Index might be diverted from its descriptive objective: “Whenever an indicator comes to be used in the reward system of an organization or institutional domain, there develop tendencies to manipulate the indicator so that it no longer indicates what it once did” [34], wrote the American sociologist in the mid-1970s, long before metrics came to govern academia. Such goal displacement, now known as Goodhart’s law (James Griesemer), led to profound transformations in the very meaning of scientific productions. Once a “knowledge unit,” the scientific paper thus became an “accounting unit,” argues Yves Gingras persuasively. We count papers and citations as if they really were objective units of measure.

The perverse effects of metrics are discernable on very different levels, as illustrated in the second part of the book, Collaborative Manipulations. Although it is tempting to approach this problem through the “bad apple” paradigm, where individual scientists found guilty of deviant behavior are used as convenient scapegoats, Barbara M. Kehm (along with several other contributors) makes it clear that the metrics problem is probably first and foremost a macro-level, institutional issue, as illustrated by university rankings. Importantly, and reflecting a wider trend, the former rankings proposed by academic specialists that focused on specific disciplines or research areas were gradually replaced by the ranking of entire institutions by the generalist press (U.S. News & World Report, The Guardian, Der Spiegel, Le Nouvel Observateur, La Repubblica, etc.). In sociological terms, metrics lessened the scientific field’s autonomy over time. This loss of autonomy is also palpable in peripheral scientific countries such as the Czech Republic, where publishing in English, loosely defined as “internationalization,” became an important criterion for research evaluation (Sarah de Rijcke and Tereza Stöckelová).

As the scope and weight of metrics were extended, new types of social actors also began to appear to counteract their undesirable effects. The third part of the book, aptly entitled Interventions: Notes from the Field, features contributions from people who, one way or another, grappled with the metrics problem. From Retraction Watch (Ivan Oransky) to PubPeer (Boris Barbour and Brandon M. Stell), to fictitious scholar with real publications and citations Ike Antkare (Ike Antkare), and fake scientists serving on the editorial boards of so-called “junk journals” (Burkhard Morgenstern), the metrics game is played variously and with different intents. Interestingly, the different cases discussed demonstrate scientists’ capacity to hijack and resist metrics, both for their personal interest and in defense of broader cultural ideals (integrity, humility, honesty, etc.).

The line is sometimes blurry between these resistances that are typically valued among academics, and the “brand appropriation” [239] and less legitimate hijackings that are analyzed in the last part of the book, Mimicry for Parody or Profit. It is sometimes delicate for social actors to determine whether or not hijackings derive from “noble,” that is non-financial, intentions. For instance, Alessandro Delfanti sheds light on viXra.org, “an ironic copycat version of the ‘official’ website” [261] arXiv. The developers of the website state that “The similarity of web design is a form of parody to highlight the endorsement and moderation policies of arXiv.org which we believe are a hindrance to scientific progress. We reverse the name and colours as a symbol of our opposing policies and to ensure that there is no confusion between the sites” [264]. Put otherwise, viXra.org can be understood as a satirical critic of arXiv’s social control over science: while promoting its virtue of openness, arXiv’s moderators do act as judges separating the wheat from the chaff.

Gaming the Metrics is a welcome first step in the investigation of performance quantification in academia. Its wide scope and the varied profile of its contributors allow for a broad exploration of the metrics problem, well beyond the classical—but nonetheless important—cases of h-index or university rankings. In the rest of this review, I will highlight two main conceptual problems and research avenues for those interested in these new kinds of deviance.

The first problem regards our conception of deviance. One fracture that appears quickly enough in this collective book is the attitude to adopt vis-à-vis so-called “predatory” journals. For some authors, we should abstain from using this term (and its derivatives) altogether, for it is accusatory rather than analytical. In fact, argues Marie-Andrée Jacob, “it is not impossible that the actual activities of counterfeiters have more in common with those of ‘real scientists’ that we can imagine” [254]. Conversely, others more or less explicitly acknowledge that we can identify which journals are predatory and which are not; there really is a boundary out there, and this boundary is relevant to the production of sound science. Hence, Finn Brunton writes that “no more than the gibberish papers, screeds, and obvious pseudoscience published by spam journals could hold up for an actual scholarly audience” [245].

This fracture begs an important question, one that is familiar to sociologists of deviance: is predation a characteristic of the journals themselves, or is it an accusation that some scientists direct toward them? Since the interactionist studies of deviance, it is generally accepted among social scientists that “social groups create deviance by making the rules whose infraction constitutes deviance, and by applying those rules to particular people and labeling them as outsiders. From this point of view, deviance is not a quality of the act the person commits, but rather a consequence of the application by others of rules and sanctions to an ‘offender.’ The deviant is one to whom that label has successfully been applied; deviant behavior is behavior that people so label”Footnote ¹.

Such a perspective necessarily reverses one’s viewpoint from “why are these journals acting predatorily?” to “how do scientists come to perceive some journals as predatory and others as ‘real’?”. What are the semiotic signs that a journal is “predatory”? It is precisely here that the fracture lies: some scholars (typically social scientists) primarily want to study the phenomenon, to understand it scientifically, while others are eager to act on it, perhaps too quickly. To be sure, these two objectives do not have to be incompatible, but putting forward the political dimension can harm the scientific aspiration, and vice versa. When Paul S. Brookes declares “If the data is whack, you must retract” [169], I, a sociologist of science, am tempted to answer back “what is whack?”. Separating “good” from “bad” science is not a transparent, clear-cut processFootnote ², and the recent controversies surrounding the use of chloroquine against Covid-19 constitute a valuable reminder that the soundness of data can be at the center of heated debates; scientific certainty is a product of time, and the dust rarely settles before an urgent need for action emerges.

Another fruitful research avenue regarding the reign of metrics in academia lies in the convergence of professional practices and symbolic hierarchies. Science is not unique in this regard, and comparisons with other professional fields where quantitative evaluations are routinely used, from policeFootnote ³ to journalismFootnote ⁴ and hospitalsFootnote ⁵, could be sociologically fruitful. For instance, in France, the increasing use of metrics by police administrations has led to phenomena that resemble what we can observe in academia. Police officers feel frustrated by what they perceive as a loss of professional autonomy; in order to counteract these pressures, they have developed a range of strategies to avoid the metrics ruleFootnote ⁶. Of course, this is not without consequences on the culturally defined quality of their work. To be in line with their supervisors’ expectations, and to avoid the hindrances that could impact their careers, some units regularly perform “easy” arrests (often targeting marginalized populations such as racial minorities) to be in line with their quantitative objective (x arrests per month) before coming back to more “serious,” long-term, costly investigations. To be sure, one should not deduce from these similitudes between police and scientific work that metrics uniformly lead to professional homogenization, as there is always room for differentiation. For instance, ethnographic research conducted among US and French journalists clearly demonstrates that the use and meaning of metrics differ from one social field to anotherFootnote ⁷. In the end, only empirical work can measure the precise effects of the use of metrics in a given professional field.

Still, if we agree that police officers’ arrests are comparable to journalists’ clicks, which are themselves akin to academics’ scientific articles as professional accounting units, then the effects of the use of metrics in science are not necessarily unique to our milieu. This calls for a more general sociology of quantification, one in which academia would be a case study among othersFootnote ⁸. In the long run, this distancing might give us a clearer view of scientific metrics, as competing working hypotheses would be confirmed or turned down along the way. For instance, the apparently opposite effects of convergence/differentiation mentioned previously might be explained away by the different scales of analysis adopted in the study of professional metrics. At a macro-level, metrics appear to lead to professional convergence, whether in science, police or journalism. But at a meso- or micro-level, some forms of differentiation could also emerge. In science, it is rather likely that metrics do not exert the same influence in every discipline and countries. French sociologists, for instance, seem to be more resistant to the government of metrics than French economists or US sociologists. Journal hierarchies, in particular, are typically not put in terms of impact factor but of historical importance and symbolic prestige. Likewise, it is not considered standard practice to mention one’s h-index on one’s curriculum vitae, and it could even be frowned upon by scholars who feel that standardization and the neoliberal ideology must be resistedFootnote ⁹. Hence, one can hypothesize that the strategies used for the gaming of metrics is largely dependent upon the subspace of the scientific field under consideration. A parallel with criminal justice would lead us to think that bureaucratic crimes might not be prosecuted and sanctioned similarly in all disciplines and countries. Ultimately, this raises the question of how serious a game science really is?