Hostname: page-component-745bb68f8f-b95js Total loading time: 0 Render date: 2025-02-11T09:17:45.180Z Has data issue: false hasContentIssue false

Making replication prestigious

Published online by Cambridge University Press:  27 July 2018

Krzysztof J. Gorgolewski
Affiliation:
Department of Psychology, Stanford University, Stanford, CA 94305. krzysztof.gorgolewski@gmail.comrusspold@stanford.eduhttp://blog.chrisgorgolewski.org/http://www.russpoldrack.org/
Thomas Nichols
Affiliation:
Oxford Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Headington, Oxford OX3 7FZ, United Kingdom. Welcome Centre for Integrative Neuroimaging, FMRIB, Nuffield Department of Clinical Neurosciences, University of OxfordOX3 7FZ, United Kingdom. Department of Statistics, University of Warwick, Coventry CV4 7AL, United Kingdom. thomas.nichols@bdi.ox.ac.ukhttps://warwick.ac.uk/fac/sci/statistics/staff/academic-research/nichols
David N. Kennedy
Affiliation:
Department of Psychiatry, University of Massachusetts Medical School, Worcester, MA 01655. david.kennedy@umassmed.edu
Jean-Baptiste Poline
Affiliation:
Montreal Neurological Institute, McGill University, Montréal, QC H3A 2B4, Canada. jbpoline@gmail.com Helen Wills Neuroscience Institute, University of California, Berkeley, CA 94720.
Russell A. Poldrack
Affiliation:
Department of Psychology, Stanford University, Stanford, CA 94305. krzysztof.gorgolewski@gmail.comrusspold@stanford.eduhttp://blog.chrisgorgolewski.org/http://www.russpoldrack.org/

Abstract

Making replication studies widely conducted and published requires new incentives. Academic awards can provide such incentives by highlighting the best and most important replications. The Organization for Human Brain Mapping (OHBM) has led such efforts by recently introducing the OHBM Replication Award. Other communities can adopt this approach to promote replications and reduce career cost for researchers performing them.

Type
Open Peer Commentary
Copyright
Copyright © Cambridge University Press 2018 

Zwaan et al.'s otherwise excellent review of issues related to replications paints a rather bleak picture of credit given to researchers involved in replication studies. Even though the status quo is described accurately – that is, replications will not be valued as much as traditional research by hiring and tenure committees – tools and interventions exist that could change this situation (“Rewarding negative results,” 2017). We recently introduced a new approach to incentivize replications and provide credit to researchers performing high-quality replications of seminal studies. Tapping into the long tradition of scientific awards, we have designed a Replication Award that can be implemented by scientific communities, journals, institutes or departments. In addition to providing a detailed protocol on how to effectively solicit submissions, score them, and announce the winner (Gorgolewski et al. Reference Gorgolewski, Nichols, Kennedy, Poline and Poldrack2017b) we have also implemented it within the Organization for Human Brain Mapping (OHBM). The first OHBM Replication Award was presented in June 2017 to Wouter Boekel for his thorough investigation of 17 brain-behaviour relationships (Boekel et al. Reference Boekel, Wagenmakers, Belay, Verhagen, Brown and Forstmann2015). This study found evidence confirming only one of the investigated relationships (correlation between real-world social network size and grey matter volume in the amygdala (Kanai et al. Reference Kanai, Bahrami, Roylance and Rees2012). No evidence for previously reported effects was found for eight relationships, and results were inconclusive for the remaining eight.

The effectiveness of the OHBM Replication Award in the context of the promotion of replications has been evaluated by a community survey (Gorgolewski et al. Reference Gorgolewski, Nichols, Kennedy, Poline and Poldrack2017a). Of 226 respondents who were familiar with the award, 49% declared that it made them more likely to perform replications in the future, and 41% declared that it made them more likely to publish replication studies. Because prizes, awards, and other honours are often listed on curricula vitae and taken into consideration during hiring and tenure decision, we hope that this award will also have a positive influence on the careers of researchers performing replications.

We look forward to more organizations adopting Replication Awards following the lead of OHBM. The key to making replication mainstream is to provide incentives for researchers and to elevate replications to the status of a first class citizen among other scientific outputs. Replication Awards can provide such incentives a form that is already widely used in the context of evaluating careers. In the future, editorial boards and reviewers should adopt more proactive policies to facilitate the publication of replication studies (similar to NeuroImage: Clinical [Fletcher & Grafton Reference Fletcher and Grafton2013]). We also hope that new publishing initiatives such as the one initiated by the OHBM this year will facilitate this change in the future.

References

Boekel, W., Wagenmakers, E.-J., Belay, L., Verhagen, J., Brown, S. & Forstmann, B. U. (2015) A purely confirmatory replication study of structural brain-behavior correlations. Cortex 66:115–33. Available at: https://doi.org/10.1016/j.cortex.2014.11.019.Google Scholar
Fletcher, P. C. & Grafton, S. T. (2013) Repeat after me: Replication in clinical neuroimaging is critical. NeuroImage: Clinical 2:247–48. Available at: https://doi.org/10.1016/j.nicl.2013.01.007.Google Scholar
Gorgolewski, K., Nichols, T., Kennedy, D. N., Poline, J.-B. & Poldrack, R. A. (2017a) Promoting replications through positive incentives. Figshare. Available at: https://doi.org/10.6084/m9.figshare.5278327.v1.Google Scholar
Gorgolewski, K., Nichols, T., Kennedy, D. N., Poline, J.-B. & Poldrack, R. A. (2017b) Replication award creation kit. Figshare. Available at: https://doi.org/10.6084/m9.figshare.5567083.v1.Google Scholar
Kanai, R., Bahrami, B., Roylance, R. & Rees, G. (2012) Online social network size is reflected in human brain structure. Proceedings of the Royal Society B Biological Sciences 279(1732):1327–34. Available at: https://doi.org/10.1098/rspb.2011.1959.Google Scholar
Rewarding negative results keeps science on track. Editorial. (n.d.). Nature 551:414. Available at: http://www.nature.com/articles/d41586-017-07325-2.Google Scholar