Zwaan et al. have provided a service to science by synthesizing the strong case for making replications mainstream. However, they have missed – and perhaps even subtly perpetuated – what may be the central underlying impediment to having a strong culture of replications: The idea that “you are your data.”
The underlying value that ought to be common among scientists is respect for the truth. Scientific theories that are not backed up by evidence – by data or logic that supports belief in a proposition or set of propositions – are simply not scientific. Moreover, a lack of concern for the truth is linked to (and, for some, necessary for) the very antithesis of science: bullshitting (Frankfurt Reference Frankfurt2005). Naturally, however, people (and presumably scientists) differ in the extent to which they are receptive to bullshit (Pennycook et al. Reference Pennycook, Cheyne, Barr, Koehler and Fugelsang2015), value logic and evidence (Ståhl et al. Reference Ståhl, Zaal and Skitka2016), and acknowledge that their beliefs could be wrong (i.e., intellectual humility [Leary et al. Reference Leary, Diebels, Davisson, Jongman-Sereno, Isherwood, Raimi, Deffler and Hoyle2017]).
If respect for the truth is the crucial underlying value that, at least, ought to bind scientists together, what implications does this have for making replications mainstream?
Zwaan et al. adroitly point out that “researchers may feel a sense of ownership of specific research findings, which can mean that failures to replicate can feel like a personal attack, one that can have implications for evaluations of their competence” (sect. 5.5, para. 3).
It is unfortunately true that some may make inferences about the value of a researcher based on a failure to replicate their work. However, scientists may overestimate the negative reputational consequences of a failure to replicate (Fetterman & Sassenberg Reference Fetterman and Sassenberg2015) as scientific reputation is based more on process than outcome (Ebersole et al. Reference Ebersole, Axt and Nosek2016b). Indeed, being subject to a failure to replicate does not provide a strong signal as to one's respect for the truth. Quite the contrary, ones' response to a failure to replicate is a substantially stronger signal. Do you care more about your reputation than the empirical result?
Unfortunately, at least based on anecdote, abject fear is often the intuitive reaction to discovering that one is to be replicated. This reaction is precisely the opposite of what it should be. For the most part, scientists research topics in which they are interested. Moreover, setting aside petty narcissism for a moment, the highest aspiration that a scientist can have is to make a meaningful impact on her field and perhaps even the outside world. Discovering that your finding is to be subject to a replication attempt should be doubly exciting, for it indicates that (a) you are to gain information about something you are interested in, and (b) your work is impactful enough to be considered worth replicating. Producing research that causes independent scientists to spend time and money pursuing is, plainly, an accomplishment.
Why, then, are replication attempts sometimes met with trepidation? As noted by Zwaan et al., failed replications are viewed as having negative reputational consequences (Bohannon Reference Bohannon2014; Fetterman & Sassenberg Reference Fetterman and Sassenberg2015). The fear may be that a failure to replicate indicates that an individual engaged in questionable research practices – or even outright fraud – to get a significant result. However, this is an untenable conclusion when based on a single or small number of observed results. Anecdotal evidence is not acceptable in our science; it should not be acceptable in our evaluation of scientists.
Only when a pattern of nonreplicable results occurs is it justifiable for a researcher's reputation to be affected. Also, even in such a case, this is because researchers who cannot produce replicable results are likely to have put concerns about their reputation and career success (in the form of publications, tenure, grants, awards, etc.) above the pursuit of science (in the form of robust research that tells us something about the world). The offending researchers only “become their data” because (with enough observations) it becomes a signal of their stance toward the truth.
It is important to point out that the replicator, too, should care more about the empirical result than the reputation of the person being replicated. A focus on “debunking” fellow scientists is guilty of the same toxic concern about reputation (and, therefore, nonprimacy of the pursuit for truth) that is needed to justify such a targeting in the first place. The fact such debunking attempts produce data about the world is not justification for perpetuating the (identity-focused) stance that may have been the source of bad data in the first place. If we want people to stop assuming that one failure to replicate will ruin their reputation, it is imperative that we do not treat failures to replicate (outside of extreme circumstances) as having reputational consequences.
Although it is perhaps not feasible for every scientist to fully buy in to the “you are not your data” mantra, it is nonetheless important to increase its aggregate influence. To this end, scientists who demonstrate a willingness to divorce themselves from their data should be celebrated (see https://lossofconfidence.com/ for a group focused on this very thing). Awards and accolades should go to scientists who, beyond having a significant influence on their respective fields, can also provide evidence of identifying with the process of science and the pursuit of truth (e.g., via dedication to open science or revision of a previous stance based on new data; see Nosek et al. [Reference Nosek, Spies and Motyl2012]). Prestigious academic positions should be given to those who do research that is both impactful and sound (a notion that seems sufficiently obvious, but that does not necessarily correspond to the selection of individuals who have successfully created a “brand”). Finally, the significance of valuing the truth should be emphasized to graduate students and future generations of scientists, particularly in cases when the relaxing of scientific values is expedient. Ultimately, making replications mainstream will be easier if scientific incentive structures begin to align with a separation of identity and data.
Zwaan et al. have provided a service to science by synthesizing the strong case for making replications mainstream. However, they have missed – and perhaps even subtly perpetuated – what may be the central underlying impediment to having a strong culture of replications: The idea that “you are your data.”
The underlying value that ought to be common among scientists is respect for the truth. Scientific theories that are not backed up by evidence – by data or logic that supports belief in a proposition or set of propositions – are simply not scientific. Moreover, a lack of concern for the truth is linked to (and, for some, necessary for) the very antithesis of science: bullshitting (Frankfurt Reference Frankfurt2005). Naturally, however, people (and presumably scientists) differ in the extent to which they are receptive to bullshit (Pennycook et al. Reference Pennycook, Cheyne, Barr, Koehler and Fugelsang2015), value logic and evidence (Ståhl et al. Reference Ståhl, Zaal and Skitka2016), and acknowledge that their beliefs could be wrong (i.e., intellectual humility [Leary et al. Reference Leary, Diebels, Davisson, Jongman-Sereno, Isherwood, Raimi, Deffler and Hoyle2017]).
If respect for the truth is the crucial underlying value that, at least, ought to bind scientists together, what implications does this have for making replications mainstream?
Zwaan et al. adroitly point out that “researchers may feel a sense of ownership of specific research findings, which can mean that failures to replicate can feel like a personal attack, one that can have implications for evaluations of their competence” (sect. 5.5, para. 3).
It is unfortunately true that some may make inferences about the value of a researcher based on a failure to replicate their work. However, scientists may overestimate the negative reputational consequences of a failure to replicate (Fetterman & Sassenberg Reference Fetterman and Sassenberg2015) as scientific reputation is based more on process than outcome (Ebersole et al. Reference Ebersole, Axt and Nosek2016b). Indeed, being subject to a failure to replicate does not provide a strong signal as to one's respect for the truth. Quite the contrary, ones' response to a failure to replicate is a substantially stronger signal. Do you care more about your reputation than the empirical result?
Unfortunately, at least based on anecdote, abject fear is often the intuitive reaction to discovering that one is to be replicated. This reaction is precisely the opposite of what it should be. For the most part, scientists research topics in which they are interested. Moreover, setting aside petty narcissism for a moment, the highest aspiration that a scientist can have is to make a meaningful impact on her field and perhaps even the outside world. Discovering that your finding is to be subject to a replication attempt should be doubly exciting, for it indicates that (a) you are to gain information about something you are interested in, and (b) your work is impactful enough to be considered worth replicating. Producing research that causes independent scientists to spend time and money pursuing is, plainly, an accomplishment.
Why, then, are replication attempts sometimes met with trepidation? As noted by Zwaan et al., failed replications are viewed as having negative reputational consequences (Bohannon Reference Bohannon2014; Fetterman & Sassenberg Reference Fetterman and Sassenberg2015). The fear may be that a failure to replicate indicates that an individual engaged in questionable research practices – or even outright fraud – to get a significant result. However, this is an untenable conclusion when based on a single or small number of observed results. Anecdotal evidence is not acceptable in our science; it should not be acceptable in our evaluation of scientists.
Only when a pattern of nonreplicable results occurs is it justifiable for a researcher's reputation to be affected. Also, even in such a case, this is because researchers who cannot produce replicable results are likely to have put concerns about their reputation and career success (in the form of publications, tenure, grants, awards, etc.) above the pursuit of science (in the form of robust research that tells us something about the world). The offending researchers only “become their data” because (with enough observations) it becomes a signal of their stance toward the truth.
It is important to point out that the replicator, too, should care more about the empirical result than the reputation of the person being replicated. A focus on “debunking” fellow scientists is guilty of the same toxic concern about reputation (and, therefore, nonprimacy of the pursuit for truth) that is needed to justify such a targeting in the first place. The fact such debunking attempts produce data about the world is not justification for perpetuating the (identity-focused) stance that may have been the source of bad data in the first place. If we want people to stop assuming that one failure to replicate will ruin their reputation, it is imperative that we do not treat failures to replicate (outside of extreme circumstances) as having reputational consequences.
Although it is perhaps not feasible for every scientist to fully buy in to the “you are not your data” mantra, it is nonetheless important to increase its aggregate influence. To this end, scientists who demonstrate a willingness to divorce themselves from their data should be celebrated (see https://lossofconfidence.com/ for a group focused on this very thing). Awards and accolades should go to scientists who, beyond having a significant influence on their respective fields, can also provide evidence of identifying with the process of science and the pursuit of truth (e.g., via dedication to open science or revision of a previous stance based on new data; see Nosek et al. [Reference Nosek, Spies and Motyl2012]). Prestigious academic positions should be given to those who do research that is both impactful and sound (a notion that seems sufficiently obvious, but that does not necessarily correspond to the selection of individuals who have successfully created a “brand”). Finally, the significance of valuing the truth should be emphasized to graduate students and future generations of scientists, particularly in cases when the relaxing of scientific values is expedient. Ultimately, making replications mainstream will be easier if scientific incentive structures begin to align with a separation of identity and data.
ACKNOWLEDGMENTS
For comments on an earlier version, I thank Nathaniel Barr, Adam Bear, Shadi Beshai, Michal Bialek, Justin Feeney, Jonathan Fugelsang, Gordon Kraft-Todd, Srdan Medimorec, Sandeep Mishra, David Rand, Paul Seli, Nick Stagnaro, and Valerie Thompson.