Construct-Driven SJT Research

The traditional SJT paradigm has focused on establishing the criterion-related validity of SJT scores (e.g., McDaniel, Morgeson, Finnegan, Campion, & Braverman, Reference McDaniel, Morgeson, Finnegan, Campion and Braverman2001), leading to heated debates on what constructs are actually measured by SJTs. Scholars have repeatedly called for the construct-driven approach to studying SJTs (e.g., Weekley & Ployhart, Reference Weekley and Ployhart2005). The construct-driven approach considers SJTs as a measurement method that can be used to measure various constructs. This approach typically entails the following steps: (a) Constructs to be measured must be clearly defined in advance; (b) items are developed to adequately sample the conceptual domain of constructs of interest, with one SJT item tapping into one construct; (c) constructs of interest are placed in the nomological network, with their relationships with other related constructs clarified; and (d) validation efforts involve empirically examining the relationships of constructs in the nomological network in terms of internal structure, convergent and discriminant validity, nomological validity, and criterion-related validity.

Using the above criteria, we have been able to locate several SJT studies conducted during the last 15 years that followed the construct-driven approach. Various constructs have been clearly defined and measured, for instance, integrity (e.g., Chen, Reference Chen2009; Meijer, Born, Zielst, & Molen, Reference Meijer, Born, Zielst and Molen2010), personal initiative (Bledow & Frese, Reference Bledow and Frese2009), emotional intelligence (Sharma, Gangopadhyay, Austin, & Mandal, Reference Sharma, Gangopadhyay, Austin and Mandal2013), and team role knowledge (Mumford, Iddekinge, Morgeson, & Campion, Reference Mumford, Iddekinge, Morgeson and Campion2008), to name just a few. Note that none of these studies have focused on measuring general domain knowledge as per Lievens and Motowidlo. This implies that the primary driving force of recent advancements of SJT research has been the adoption of the construct-driven approach rather than Lievens and Motowidlo's suggested method.

Development of an SJT for Integrity in the Chinese Context

In this section we briefly share our own experience of developing an SJT for testing integrity in the Chinese context using the construct-driven method (see Chen, Reference Chen2009, for details). With the rapid development of Chinese economy, there has been a pressing need for assessing the integrity of middle/high-level managers and employees holding key positions in organizations. Unfortunately, most Western integrity tests have targeted entry-level employees with straightforward counterproductive work behaviors, such as stealing, and have not taken the social desirability (faking) issue seriously. Thus, these Western integrity tests were judged unable to address our need. Integrity-related behaviors do not exist in vacuum and can only be observed in specific situations that comprise a potential conflict of interests (Kaptein, Reference Kaptein1999). Given that, we felt that an SJT should be a suitable measurement method in that it can utilize various conflicting situations to solicit integrity-relevant behaviors. Therefore, we developed a new SJT to measure integrity from the bottom up.

We began by using the grounded theory approach to delineate the conceptual domain of integrity in Chinese culture and identified three dimensions of Chinese integrity within business settings. The first dimension, honesty and responsibility, refers to the extent to which an individual may engage in honest behaviors and refrain from dishonest, self-interested behaviors when his/her personal interests are in conflict with the organization's interests. The second dimension, regulation compliance, refers to the extent to which an individual obeys the rules and regulations of the organization or, in other words, being dependable and trustworthy. The third dimension, justice and fairness, refers to the extent to which an individual will follow the general ethic principles when dealing with various constituents when these constituents’ interests are in conflict with the organization's interests.

Next, we interviewed and surveyed around 80 professionals and middle/high-level managers and obtained 78 critical incidents. A group of subject matter experts (SMEs) composed of human resource directors and doctoral students in applied psychology then categorized these critical incidents into one of three integrity dimensions and then developed 30 pilot SJT items with 10 items measuring each integrity dimension. Another group of SMEs assessed the representativeness, degree of dilemma, and job relevance of these pilot items and rated the effectiveness of response options. Several pilot studies were conducted, with problematic items modified or removed, resulting in the final version of the 10-item SJT.

To alleviate the social desirability concern, we asked test takers to rank order the effectiveness of all response options in each item. SJT items were scored based on the distance between test takers’ rank order profiles and SMEs’ mean rank order profile. Many test takers commented that the task of rank ordering response options was very interesting, and they were immersed in this task during the test.

Several validation studies were conducted to assess the psychometric properties of this integrity SJT. Results showed that correlations of SJT item scores consistently yielded a clear-cut, three-factor structure corresponding to the three hypothesized integrity dimensions. SJT total scores were found to be significantly and modestly correlated with Conscientious scores (r = .15) and Openness to Experience scores (r = −.15), suggesting that integrity was a unique construct that did not overlap substantially with personality traits. SJT scores demonstrated strong correlations with supervisor-rated trustworthiness scores: rs = .46, .36, .28, and .50 for honest and responsibility, regulation compliance, justice and fairness, and total SJT scores, respectively. We also compared integrity SJT total scores among incumbent employees, professional graduate students who had prior work experience, and undergraduate students with no prior work experience. Two findings emerged. First, there was no significant group mean difference between incumbent employees and professional graduate students, suggesting our integrity SJT was resistant to faking. Second, incumbent employees and professional graduate students had significantly higher SJT total scores than did undergraduate students, implying the importance of work experience, that is, specific domain knowledge.

Therefore, our own experience of developing an SJT for integrity in the Chinese context reinforces the utility of the construct-driven approach to SJT research. It also provides indirect evidence supporting the role of specific domain knowledge in affecting SJT scores.

Reflection on the Role of Situations in SJTs

On the basis of our reviewing of SJT research and self-reflections, we contend that there are at least three reasons to believe that situations are an indispensable feature of SJTs. First and most important, although recent SJT research reviewed by Lievens and Motowidlo convincingly demonstrated that general domain knowledge can predict job performance, let us not forget that empirical evidence also has clearly shown that general domain knowledge and job specific knowledge contribute independently to predicting job performance (e.g., Motowidlo & Beier, Reference Motowidlo and Beier2010). Bledow and Frese (Reference Bledow and Frese2009) also found that both situated behavioral preferences for personal initiatives measured by an SJT and personal initiative scores based on a Likert-type of scale complemented each other to predict supervisor-rated performance. In other words, dismissing situations may lead to the loss of predictive information, which will not be well received in field settings.

Second, SJTs with specific situations may be perceived as more realistic and more job relevant and thus may have higher face validity and more favorable test-taker perceptions than SJTs without specific situations (Lievens & Motowidlo). Third, in certain SJTs such as those measuring integrity and morality, it is necessary to solicit relevant behaviors/judgment through specific situations. As cogently pointed out by Kaptein (Reference Kaptein1999), integrity is relative behaviors that manifest only through choices in specific conflicting scenarios. Similarly, when designing an SJT for emotional intelligence, Sharma et al. (Reference Sharma, Gangopadhyay, Austin and Mandal2013) wrote that understanding the contexts in which emotions are displayed is a crucial dimension in measuring this construct.

To conclude, we believe that the construct-driven approach should be the gold standard for SJT research moving forward and that the unique value of an emphasis on measuring general domain knowledge has yet to be established. Further, specific situations in SJTs are probably not going away any time soon. To summarize our view in one statement, clearly defined constructs and specific situations are the currency of SJTs.