In his compelling article, Cesario offers a cogent critique of “the widespread use of experimental social psychology to understand real-world group disparities” (abstract). In our reading, Cesario offers both narrow and broad arguments. We concur with the narrow version, which highlights three “fatal flaws” in standard experimental bias studies that undermine their direct contribution to explaining real-world group disparities in social outcomes. This critique does not imply that these studies have no value – we think they do – or that stereotype biases do not exist – of course they do, but rather that experimental evidence of biased associations do not illuminate major causes of group disparities because of a number of limitations clearly outlined in Cesario's article.

Chief among these limitations is what Cesario calls “the problem of missing information.” In contrast to these experiments, in the real world, decision-making does not operate in an informational vacuum. The strength of experiments is their control – isolating the effects of one variable by creating an informational vacuum (in this case, only social category membership). Yet these situations – devoid of individual, situational, and contextual information and with time pressures imposed to prevent the activation of conscious processing – are precisely when stereotypes (negative and positive) are relied upon to fill gaps in information. Such stereotypes are influenced not only by media hype and personal experiences, but also, in some cases, knowledge of group average behavioral differences. Thus, the strength of experiments is a weakness when extrapolating to real-world decision-making where stereotypes may not be activated given the wealth of other contextual information. This, Cesario argues and we agree, not only makes the external validity of the tests questionable, but it renders the null hypothesis of “no difference” potentially unrealistic – or at least requiring justification – given the second critique identified by Cesario – that of missing forces (which we also view as missing information).

Cesario's important critique about missing information is usefully extended to other common methods of measuring biases, including increasingly pervasive self-report discrimination measures, which can suffer from similar limitations, albeit in reverse form. In “experimental task” situations decision-makers only have group membership information. Conversely, in real life, people in interaction differ on many dimensions. In self-report discrimination instruments, individuals are asked to attribute causes of perceived unfair treatment by others usually without knowledge of intent. Individuals may attribute one cause (sex, race, weight, age, etc.) when it may be a different one (appearance, tattoos, and accent), or the perpetrator is grumpy, tired, and treating everyone poorly. This is not to suggest that all discriminatory acts are ambiguous in their source or motivation (e.g., calling someone a racist, sexist, or homophobic slur), only that much captured in self-report discrimination measures is based on attributions without full information. This reliance on perceived intent and attribution distorts measurement to some unknown extent. For example, African American women generally self-report less racial discrimination experiences than Black men (although this varies across specific discrimination type, see Burt & Simons, Reference Burt and Simons2015). This is no doubt due, in part, to ambiguity in attribution of the source of mistreatment. Focusing on sex and race (to make a point), if an African American woman is given much worse service than a white man in front of her, it could be about race, sex, or both, whereas for a Black man, it is about race (see Essed, Reference Essed1991). We note this limitation both to recognize the pervasive problem of missing information in bias research (and in many domains) as well as the fact that these experimental bias studies are not unique in their missing information problem. Note that this critique does not imply that perceptual measures are useless – indeed, we think they can be quite useful – only that the limitations should be recognized and addressed whenever possible with methodological innovation and triangulation of methods.

While we concur with what we view as the narrow version of Cesario's argument – that these studies do not identify causes of group disparities, we disagree with the broader critique – that the entire enterprise of using experimental social psychology to shed light on group disparities should be abandoned. Rather than abandonment, we suggest targeted utilization where research can use simpler models to identify stereotypes as a starting point for understanding causes of group disparities to be addressed in more comprehensive investigations (e.g., Johnson, Cesario, & Pleskac, Reference Johnson, Cesario and Pleskac2018). Understanding what stereotypes or implicit biases persist, when these influence attributions and decision-making (e.g., under cognitive load, lack of information, and high threat), and what situational, contextual, and individual information mitigates against the reliance on unconscious stereotypes remains an important research question to which these studies can contribute.

Finally, we note that we are unsure whether the issues Cesario raises are appropriately directed at experimental social psychologists. In our reading, most scholars are cautious in their claims about what these studies can tell us about causes of group disparities. Indeed, the Moss-Racusin, Dovidio, Brescoll, Graham, and Handelsman's (2012) article used as a prototypical example by Cesario was replete with cautionary statements such as “might,” “could,” “possibly,” and with explicit acknowledgement “that various lifestyle choices likely contribute to the gender imbalance in science” (p. 16764), among other caveats. We have seen journalists, activists, and scholars in other domains misrepresent these studies as identifying “major causes” of group disparities, and they would benefit from heeding Cesario's cogent analysis.

In sum, we concur with Cesario's critique about the limits of these experimental studies for identifying major causes of real-world group disparities. We also agree with Cesario that these studies can provide “important information about stereotyping processes.” Rather than final answers, we view them as valuable starting points for identifying biases that may influence decisions and, thus, disparities in certain circumstances (limited information, high cognitive load, and time pressures). Follow up work is needed to understand when, where, and how these may influence outcomes, considering full information, contingencies, and behavioral differences. All models are imperfect, and scientists must rigorously and continuously evaluate the validity of models and methods to identify limitations and flaws, making necessary improvements and corrections, especially when findings have social implications.