Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-02-06T10:52:13.494Z Has data issue: false hasContentIssue false

The “‘Crisis’ Crisis” in psychology

Published online by Cambridge University Press:  10 February 2022

John D. Medaglia
Affiliation:
Department of Psychological and Brain Sciences, Drexel University, Philadelphia, PA, 19104, USAjdm582@drexel.eduwww.cognew.com Department of Neurology, Drexel University, Philadelphia, PA, 19104, USA Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA 19119, USA
Kiante A. Fernandez
Affiliation:
Department of Psychology, Ohio State University, Columbus, OH43210, USA. kaf395@drexel.edu

Abstract

The recent trend to label dilemmas in psychology as “crises” is insidious. The “‘Crisis’ Crisis” in psychology can distract us from actionable practices. As a case in point, “The Generalizability Crisis” offers the valuable central thesis that verbal-quantitative gaps imperil psychological science. Focusing on the key issues rather than crisis narratives can lead to progress in our discourse and research.

Type
Open Peer Commentary
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press

“The Generalizability Crisis” offers lucid insights into the problems that can occur when researchers inappropriately use statistical models to test hypotheses and generalize their findings. However, labeling trends in psychological science as “crises” is a new form of insidious professional communication. Authors can convey a sense of immediacy, severity, and unity of causes and effects that are sensationalizing at best and counterproductive at worst. Collective action could mitigate the trends that ultimately lead authors to write “crisis” articles. Psychologists have possessed the intellectual and quantitative tools to prevent these dilemmas in psychological science for many years. We should expect that people will often fail to apply these tools (Lilienfeld, Reference Lilienfeld2017). Furthermore, we should not expect powerful new analyses to automatically solve all problems (Kell & Oliver, Reference Kell and Oliver2004; Shneiderman, Reference Shneiderman2016). With the intent to halt an emerging rhetorical practice, we concisely call attention to the “‘Crisis’ Crisis” in psychology, its possible causes, and how to obviate it.

Crises are turning points: unstable or critical moments in which a decisive change is impending (Merriam-Webster, n.d.), often associated with difficulty, danger, and suffering (Dictionary, Reference Dictionaryn.d.). It is not clear that this term best describes the concerns offered by Yarkoni, nor the preceding and ongoing replication crisis (Maxwell, Lau, & Howard, Reference Maxwell, Lau and Howard2015) or reproducibility crisis (Baker, Reference Baker2016; Open Science Collaboration, 2015). Recently, crisis narratives have been uncritically endorsed by most authors writing about the topic, whereas the trends in question might be better characterized as epochal change (Fanelli, Reference Fanelli2018). With the increased availability of meta-research practices (Lakens, Hilgard, & Staaks, Reference Lakens, Hilgard and Staaks2016; Soderberg et al., Reference Soderberg, Errington, Schiavone, Bottesini, Thorn, Vazire and Nosek2020), transparent and open data sharing and preregistration (Nosek, Ebersole, DeHaven, & Mellor, Reference Nosek, Ebersole, DeHaven and Mellor2018), and instant and accessible communication technologies, enhanced visibility and discussion of undesirable practices could be a harbinger of positive change, not a crisis to be averted (Nosek et al., Reference Nosek, Hardwicke, Moshontz, Allard, Corker, Almenberg and Vazire2021). By espousing crisis narratives, we should be mindful of the risk of contributing to bandwagoneering negativity, cynicism, indifference, and antiscientific sentiments (Fanelli, Reference Fanelli2018). Considered unironically, the “‘Crisis’ Crisis” in psychology is no crisis at all if we identify its causes and obviate further alarm.

An unfortunate victim of the “‘Crisis’ Crisis” is one of Yarkoni's excellent central theses. We could not agree more that the gulf between verbal statements and inferential statistics (or any quantitative concept) can impede progress in psychology. On that basis, we fully agree with Yarkoni's suspicion that the verbal-quantitative divide in psychology is one of its fundamental challenges. Unfortunately, this long overdue critique is wrapped within the narrative-reinforcing guise of a crisis, which takes some of the emphasis off of the key argument. Nontrivially, the verbal-quantitative divide is philosophically and temporally antecedent to any specific concerns about generalizability. The verbal-quantitative divide is not unique to issues that cause or contribute to problems that arise when psychologists aim to generalize their findings. Nor is it isolated to issues inherent in applying linear mixed models to psychological and behavioral data, or other cases that Yarkoni considers. This basic point is so significant that we were motivated to write this response article to urge authors to eschew “crisis” overtones. In this case, we should instead focus on a more fundamental issue that affects much of psychological science: too often, what we say we study does not match what we do quantitatively. That is a big problem.

How did we arrive at this crisis of all crises in psychology? Authors often respond to trends that have a basis in facts, but the tendency to call them “crises” is worth a moment's reflection. If crisis narratives are self-reinforcing with relatively few published or conversational counterpoints, they may be trivially self-sustaining. As others have noted, the concern that research quality is declining is neither new nor universally justified (Fanelli, Reference Fanelli2018), and calling attention to misapplied concepts and methods is part of the routine business of science (Gelman & Loken, Reference Gelman, Loken and Pitici2016). Perhaps psychologists are alarmed when they notice patterns and trends that they did not perceive before due to our increased focus on meta-research. Others could be concerned that psychological science is perverse incentives or bad actors all the way down (Lilienfeld, Reference Lilienfeld2017). In either case, scientists have created the tools to identify, evaluate, and communicate about these trends, call attention to them, and propose solutions. Perhaps the perceived value of calling for a virtuous change in academic psychology has increased such that some authors spend time writing about it (Whitaker & Guest, Reference Whitaker and Guest2020). Labeling calls to change a “crisis” could receive more press, views, and citations, which can be potent reinforcers for authors and editors (Dworkin et al., Reference Dworkin, Linn, Teich, Zurn, Shinohara and Bassett2020; King, Bergstrom, Correll, Jacquet, & West, Reference King, Bergstrom, Correll, Jacquet and West2017; Moed et al., Reference Moed, Colledge, Reedijk, Moya-Anegon, Guerrero-Bote, Plume and Amin2012; Ruscio, Reference Ruscio2016).

Systemic problems call for collective and individual actions. To identify and absolve “crisis”-driving issues, we point readers to copious writing about incentives and other structural issues including salary composition (Bourne, Reference Bourne2018), research funding dynamics (Lilienfeld, Reference Lilienfeld2017; Wahls, Reference Wahls2018), citation practices (Stephan, Reference Stephan2012), and individual researcher choices (Chambers, Reference Chambers2017). Special attention to reinforcing best standard practices for established and new quantitative practices is essential (Amrhein, Trafimow, & Greenland, Reference Amrhein, Trafimow and Greenland2019; Loken & Gelman, Reference Loken and Gelman2017; Shrout & Rodgers, Reference Shrout and Rodgers2018; Trafimow, Reference Trafimow2018). Many issues would be resolved by adhering to practices we teach to students (Chopik, Bremner, Defever, & Keller, Reference Chopik, Bremner, Defever and Keller2018). Others would be resolved by consulting quantitative domain experts who can help bridge verbal-quantitative gaps. In agreement with Yarkoni, we encourage readers to ask if a concept is quantifiable, and if so, find a procedure to sufficiently test the idea. We should always ask what we aim to study, how to measure it, and what we can and cannot conclude from our experimental and quantitative methods. If we must consider a qualitative method, we should still justify it and consider what quantification could benefit us. It could be that psychologists generate many verbalizable concepts that are invalid and worth no further pursuit. But we should remember that most sciences have been in this predicament at some point in time.

Financial support

JDM acknowledges support from NIH grants DP5-OD-021352-01, R01-DC-16800-01A1, R01-DC-014960-01A1, R01-AG-059763, and Department of the Army grant PRMRP 12902164.

Conflict of interest

None.

References

Amrhein, V., Trafimow, D., & Greenland, S. (2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don't expect replication. The American Statistician, 73(S1), 262270.CrossRefGoogle Scholar
Baker, M. (2016). Reproducibility crisis. Nature, 533(26), 353366.Google Scholar
Bourne, H. R. (2018). Opinion: Expansion fever and soft money plague the biomedical research enterprise. Proceedings of the National Academy of Sciences, 115(35), 86478651.CrossRefGoogle ScholarPubMed
Chambers, C. (2017). The seven deadly sins of psychology. Princeton University Press.Google Scholar
Chopik, W. J., Bremner, R. H., Defever, A. M., & Keller, V. N. (2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45(2), 158163.CrossRefGoogle Scholar
Dictionary, C. (n.d.). Crisis. In dictionary.cambridge.org dictionary. Retrieved February 13, 2020 from https://dictionary.cambridge.org/us/dictionary/english/crisis.Google Scholar
Dworkin, J. D., Linn, K. A., Teich, E. G., Zurn, P., Shinohara, R. T., & Bassett, D. S. (2020). The extent and drivers of gender imbalance in neuroscience reference lists. Nature Neuroscience, 23(8), 918926.CrossRefGoogle ScholarPubMed
Fanelli, D. (2018). Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences, 115(11), 26282631.CrossRefGoogle ScholarPubMed
Gelman, A., & Loken, E. (2016). The statistical crisis in science. In Pitici, M. (Ed.), The best writing on mathematics (pp. 305318). Princeton University Press.Google Scholar
Kell, D. B., & Oliver, S. G. (2004). Here is the evidence, now what is the hypothesis? The complementary roles of inductive and hypothesis-driven science in the post-genomic era. Bioessays, 26(1), 99105.CrossRefGoogle ScholarPubMed
King, M. M., Bergstrom, C. T., Correll, S. J., Jacquet, J., & West, J. D. (2017). Men set their own cites high: Gender and self-citation across fields and over time. Socius, 3, 2378023117738903.CrossRefGoogle Scholar
Lakens, D., Hilgard, J., & Staaks, J. (2016). On the reproducibility of meta-analyses: Six practical recommendations. BMC Psychology, 4(1), 110.CrossRefGoogle ScholarPubMed
Lilienfeld, S. O. (2017). Psychology's replication crisis and the grant culture: Righting the ship. Perspectives on Psychological Science, 12(4), 660664.CrossRefGoogle Scholar
Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584585.CrossRefGoogle ScholarPubMed
Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487.CrossRefGoogle ScholarPubMed
Merriam-Webster. (n.d.). Crisis. In Merriam-webster.com dictionary. Retrieved February 13, 2020 from https://www.merriam-webster.com/dictionary/crisis.Google Scholar
Moed, H. F., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A., & Amin, M. (2012). Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics, 92(2), 367376.CrossRefGoogle Scholar
Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 26002606.CrossRefGoogle ScholarPubMed
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Almenberg, A. D., … Vazire, S. (2021). Replicability, robustness, and reproducibility in psychological science. Retrieved from https://psyarxiv.com/ksfvq/.Google Scholar
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).Google Scholar
Ruscio, J. (2016). Taking advantage of citation measures of scholarly impact: Hip hip h index!. Perspectives on Psychological Science, 11(6), 905908.CrossRefGoogle ScholarPubMed
Shneiderman, B. (2016). Opinion: The dangers of faulty, biased, or malicious algorithms requires independent oversight. Proceedings of the National Academy of Sciences, 113(48), 1353813540.CrossRefGoogle ScholarPubMed
Shrout, P. E., & Rodgers, J. L. (2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology, 69, 487510.CrossRefGoogle ScholarPubMed
Soderberg, C. K., Errington, T., Schiavone, S. R., Bottesini, J. G., Thorn, F. S., Vazire, S., … Nosek, B. A. (2020). Research quality of registered reports compared to the traditional publishing model. Retrieved from https://osf.io/aj4zr/.Google Scholar
Stephan, P. (2012). Perverse incentives. Nature, 484(7392), 2931.CrossRefGoogle ScholarPubMed
Trafimow, D. (2018). An a priori solution to the replication crisis. Philosophical Psychology, 31(8), 11881214.CrossRefGoogle Scholar
Wahls, W. P. (2018). Point of view: The NIH must reduce disparities in funding to maximize its return on investments from taxpayers. Elife, 7, e34965.CrossRefGoogle ScholarPubMed
Whitaker, K., & Guest, O. (2020). #bropenscience is broken science: Kirstie Whitaker and Olivia guest ask how open “open science” really is. The Psychologist, 33, 3437.Google Scholar