Yarkoni presents the psychologist with a choice: either embrace qualitative analysis or else adopt the specific solutions described in section 6.3 and suffer the consequences. Yet the initial choice itself is a false dilemma and the solutions are half-measures. The rub is that Yarkoni's proposals are based on three dubious assumptions: (1) that empirical science is only about collecting and analyzing data, which leaves theoretical investigation almost completely out of the picture, (2) that the distinction between quantitative and qualitative research is fundamental to addressing the crisis, and (3) that qualitative research is, essentially, quantitative inquiry sans inferential statistics.

As to assumption (3), qualitative investigation differs from quantitative research in many important respects, determined in the final analysis by the different aims the two kinds of inquiry are intended to achieve. Qualitative researchers rely on in-depth interviews, observation, and document analysis rather than experimentation. Also, the logic behind purposeful sampling used in qualitative research is, roughly speaking, a mirror image of the logic behind statistical sampling: what is a strength in one would be a weakness in the other (Patton, Reference Patton, Everitt and Howell2005). Therefore, Yarkoni is wrong when he says that, in many subfields of psychological science, embracing qualitative analysis would amount to merely dropping statistical inference. That would lead to replacing one kind of poor quality research by another, and doing poor quality research is no answer to any crisis. Correspondingly, while we have nothing against good qualitative inquiry, abandoning quantitative research in favor of it would amount to throwing the baby out with the bathwater. The two approaches are not mutually exclusive but complementary (Shadish, Reference Shadish1993). Psychology needs both.

So, to address the generalizability crisis, we must fix the way quantitative research is done. The question is: how? Here, Yarkoni's assumptions (1) and (2) intervene by distorting and restricting available options. Interestingly, although assumption (1) blinds Yarkoni to the significance of theoretical work in general, he wouldn't have been able to make his case without appealing to theoretical insights. He resolves the tension by conflating theorizing with qualitative analysis. This is a mistake. Theorizing is an activity integral to any scientific approach regardless of its specific aims and methods. It transcends the difference between qualitative and quantitative research. This, we believe, is crucial because all the shortcomings of current practice discussed by Yarkoni come from a common source: researchers' inadequate appreciation of how various theoretical considerations should inform the decisions made at every stage of scientific investigation.

Consider Yarkoni's critique of Alogna et al.'s (Reference Alogna, Attaya, Aucoin, Bahník, Birch, Birt and Zwaan2014) many-lab replication of verbal overshadowing. From our perspective, it showcases that choices regarding which phenomena to study empirically, or which results to replicate, ought to be made against a broader theoretical background that includes general theoretical principles such as “the most basic, uncontroversial facts about the human mind” cited by Yarkoni. Ideally, such principles should be systematized to provide insights into the human mind, forming a theoretical framework to guide further research (Irvine, Reference Irvine2021; Muthukrishna & Henrich, Reference Muthukrishna and Henrich2019). But they should not be ignored even in the absence of such a framework.

The generalizability crisis, by contrast, is more to do with local theories. It arises from widespread failures to appreciate the relations between the rich conceptual variables employed by micro- or middle-range theories (Cartwright, Reference Cartwright2020), on one hand, and their operationalizations, on the other (cf. Shadish, Cook, & Campbell, Reference Shadish, Cook and Campbell2002). These failures often yield inconsistencies between data and various components of accepted theory, understood as a coherent and well-ordered set of concepts or models. While Yarkoni rightly draws attention to the mismatch between verbal and mathematical descriptions, in many cases, the inconsistencies can be understood without special statistical training – for example, the nature and possible ramifications of mono-operation and mono-method biases are easy to comprehend (Shadish et al., Reference Shadish, Cook and Campbell2002, 75–76).

To tie Yarkoni's critique of Alogna et al. (Reference Alogna, Attaya, Aucoin, Bahník, Birch, Birt and Zwaan2014) with the generalizability crisis, note that inattention to theory can affect generalizability even if some of the resulting issues are not directly related to operationalization. For example, attempts to integrate research associated with mutually inconsistent theoretical frameworks can cause confusion and thereby affect validity across the board – as has recently been observed in research on emotion (Weidman, Steckler, & Tracy, Reference Weidman, Steckler and Tracy2017).

The moral we draw is that the generalizability crisis is unlikely to go away as long as the community as a whole does not recognize that an adequate understanding of theory is essential to research validity. It is theory, not gut feelings, that should tell us what kind of data to collect, how to collect them, and how to analyze and interpret them (Kukla, Reference Kukla1989). Furthermore, pace Yarkoni, theory cannot be read off of empirical data: theory needs to be developed, which requires a set of skills different from that of the experimenter. Like many other sciences, psychology needs specialized theorists whose work visibly contributes to experimental research (MacKay, Reference MacKay1988).

Let us close by calling attention to important similarities between the generalizability crisis and the replication crisis. Both have been with us for quite some time and both involve widespread violation of fundamental and well-known principles of scientific investigation. It is fairly obvious, for example, that the findings of a single small study may very well be false positives, especially after some p-value hacking. It is equally obvious that the inferences we draw from obtained data should be warranted. Arguably, researchers do not need Yarkoni to educate them about the need for conservative conclusions: they know the rules – they just do not follow them. This suggests that we should explore measures focused on changing the research culture (Nosek, Spies, & Motyl, Reference Nosek, Spies and Motyl2012). But although many practices advocated by the open science movement, such as data sharing and improved quality of reporting (Hensel, Reference Hensel2020; Miłkowski, Hensel, & Hohol, Reference Miłkowski, Hensel and Hohol2018), can help to enhance both reproducibility and generalizability (the latter, by enabling high-quality re- and meta-analysis), it is also necessary to strengthen theorizing and work toward consistently incorporating theoretical results into experimental research. Without that, psychology will be a headless rider doomed to face ever new crises.