Although the authors make strong arguments for both sides of the debate in “Getting Rid of Performance Ratings: Genius or Folly? A Debate,” we argue that performance appraisal reactions were largely overlooked beyond a few exceptions, where the authors either alluded to or explicitly mentioned reactions. For example, the authors explain that one reason organizations have eliminated the forced distribution approach is negative employee reactions. The authors also highlight the importance of managers using appropriate language when delivering performance appraisal ratings in order to improve employee reactions. Despite these exceptions, we believe it is necessary to call more attention to the critical role of ratee reactions in the performance appraisal process. Therefore, our commentary expands on the conversation sparked by Adler et al. (2016) by incorporating ratee reactions.

The 12 papers commenting on K. R. Murphy (2009a) raise a number of important issues, most of which can be subsumed in one of four themes. First, papers examining content-oriented validation strategies are still necessary and useful, in part because of the frequent use of these strategies in the practice of industrial–organizational (I–O) psychology. Second, the term “content validity” means many different things both within and beyond the field of I–O psychology, and it is useful to understand what sorts of inferences examinations of test content do and do not support. Third, these 12 papers present very little evidence that content validation, as typically carried out by I–O psychologists, actually provides information about the likelihood that people who do well on the test will do well on the job. Finally, I believe that the best use of content-related evidence in validating selection tests is in developing hypotheses about relationships between test scores and criteria rather than in testing these hypotheses.

The results of meta-analytic (MA) and validity generalization (VG) studies continue to be impressive. In contrast to earlier findings that capped the variance accounted for in job performance at roughly 16%, many recent studies suggest that a single predictor variable can account for between 16 and 36% of the variance in some aspect of job performance. This article argues that this “enhancement” in variance accounted for is often attributable not to improvements in science but to a dumbing down of the standards for the values of statistics used in correction equations. With rare exceptions, applied researchers have suspended judgment about what is and is not an acceptable threshold for criterion reliability in their quest for higher validities. We demonstrate a statistical dysfunction that is a direct result of using low criterion reliabilities in corrections for attenuation. Corrections typically applied to a single predictor in a VG study are instead applied to multiple predictors. A multiple correlation analysis is then conducted on corrected validity coefficients. It is shown that the corrections often used in single predictor studies yield a squared multiple correlation that appears suspect. Basically, the multiple predictor study exposes the tenuous statistical foundation of using abjectly low criterion reliabilities in single predictor VG studies. Recommendations for restoring scientific integrity to the meta-analyses that permeate industrial–organizational (I–O) psychology are offered.

Although there is no doubt that Likert scaling suffices for straightforward scale development and use, it is important to appropriately model the response process for more complex measurement problems. In this response, we comment on the response process and four applications: assessment of dimensionality, computerized adaptive testing, differential item functioning, and individual differences in responding. In each case, we argue that correctly modeling the psychology of responding is critical.

As researchers who have advanced victim precipitation arguments in our own work on victimization and job performance (Jensen, Patel, & Raver, 2014), we agree fully with this statement made by Cortina, Rabelo, and Holland (2018): “A victim's traits or behaviors might help us understand why the instigator chose that particular person for abuse, but we must always emphasize that it was the instigator, not the victim, who did the choosing and abusing” (p. 93). An overemphasis on victim characteristics does deflect attention away from wrongdoers, and theory that encourages us to consider perpetrator motivation and the social environment is needed.