Now we turn to the four potential ways identified by the focal authors that worker sample underrepresentation could influence the results of organizational research. Our focus with this examination remains with the selection domain.
1. Overlooked Phenomena
There are two key reasons why potentially important phenomenon could be overlooked in selection of workers. First, given their higher prevalence (and sometimes importance), the operationalization of constructs in professional samples could carry over into worker samples even if the construct takes on different meaning in the latter. For instance, a job analyst highly attuned to developing selection systems for professional positions might also operationalize flexibility for worker positions more in terms of learning and knowledge even though variations in shift scheduling may be more crucial.
Second, there appears to be somewhat of a general tendency across occupations for job analysts to focus more strongly on knowledge, skills, and abilities (i.e., KSAs) than on the other (i.e., “O”) aspects of a position. A predominant focus on KSAs may suffice for professional positions where KSAs collectively tend to be more prevalent and important than other characteristics. But, for some (perhaps many) worker occupations, the other characteristics can dominate in terms of both occurrence and importance. For instance, schedule flexibility can be critical with some worker positions (e.g., fast food), and aspects such as willingness to stay late, cover additional shifts on short notice, and rotate assigned shifts (e.g., first vs. second) should all fall under “O.”
Continuing, one potential result of economic tenuousness, job insecurity, and holding second jobs could be less tenure. The process of replacing an employee is time consuming and costly even for the lowest level positions. Given the propensity for high turnover in a number of worker occupations (e.g., 100% or more is common in retail and fast food), it would seem logical for selection processes to include some elements designed to identify individuals who are more likely to stay a reasonable length of time with the organization.
To provide at least a partial assessment of whether flexibility and tenure are being overlooked in worker samples, we picked four among our database of 27 worker samples that were highly likely to involve schedule instability and have high turnover. They included clothing sales associates (Roth, Van Iddekinge, Huffcutt, Eidson, &Schmitt, Reference Roth, Van Iddekinge, Huffcutt, Eidson and Schmitt2005); home improvement store employees (Little, Schoenfelt, & Brown, Reference Little, Schoenfelt and Brown2000); appliance factor service technicians (Delery, Wright, McArthur, & Anderson, Reference Delery, Wright, McArthur and Anderson1994); and discount clothing, accessories, and home products sales associates (Motowidlo, Brownlee, & Schmit, Reference Motowidlo, Brownlee and Schmit2008).
Clothing sales associate. The main selection mechanism described in this study was a highly structured interview, although a Big Five personality test was also administered. There were six interview questions, designed respectively to assess (a) helping customers pick a gift, (b) helping customers match clothes, (c) helping when off the clock, (d) dealing with customers who ask for a discount, (e) identifying dishonest customers, and (f) dealing with an upset customer. The majority of these six content areas appear to be focused on the core KSA elements of the retail sales environment. Only one (helping when off the clock) appears to relate to schedule instability, although only to staying later and not variability in shift scheduling (e.g., start/end time) and/or covering additional shifts on short notice. None of the dimensions appear to address tenure, although that could have been covered via other selection mechanisms such as looking for “job-hopping” in the résumé.
Regarding the personality test, the primary purpose of its administration appears to have been assessing personality saturation in the interview questions. Regardless, to assess flexibility and tenure might have required a stronger focus on specific facets of the Big Five. For instance, impulsiveness might be the most relevant facet of neuroticism for tenure, while action might be the most relevant facet of openness for flexibility.
Home improvement store employees. A job analysis identified five core dimensions: (a) initiative, (b) problem solving, (c) communication/interpersonal skills, (d) quality, and (e) product knowledge. We note that schedule flexibility is not included as a dimension. Nor is there anything related to tenure, although again it might have been addressed elsewhere in the selection process.
Factory service technicians. A job analysis identified two main clusters, one for technical prowess and the other for interpersonal skills/customer service orientation. Both mental ability tests and a structured interview were included in their selection battery. Here there is a strong focus on mental/technical skills, which is not surprising given that appliances can malfunction in a variety of different ways, and the ability to diagnose problems and identify solutions quickly is a distinct advantage. The focus on customer interaction is also not surprising given that quick diagnoses and repair often cannot compensate for poor customer skills. What we do not see in this study is a reference to schedule flexibility, which we presume is common in this job area. For instance, the last job of the day may take considerably longer than anticipated, thereby requiring the technician to stay later than the target quitting time.
Discount clothing, accessories, and home products sales associates. Here the job analysis itself focused directly on customer interaction. From this analysis, a selection battery containing an intelligence test, a personality test, several roleplays, and a structured interview was developed. Selection content oriented toward the workers themselves (e.g., flexibility, tenure) appears to be absent.
In summary, these four studies suggest the possibility that at least some phenomenon unique to (or at least more strongly associated with) worker occupations could be overlooked. Only one of these four studies included flexibility in scheduling as a dimension, and even then, it was limited. A common element across these studies was the use of a critical incident job analysis. It is possible that the individuals contributing critical incidents tend to think of issues pertaining directly to the work itself (e.g., knowledge/learning, customer contact) rather than tangential issues such as scheduling and reducing turnover. Alternatively, it is possible that at least some critical incidents related to more tangential issues emerge, but the job analysts choose to focus more on those pertaining directly to core work issues (i.e., the KSAs).
3. Worker Status
As evidence for differences in worker status, Bergman and Jean noted that GMA has been found to differentially predict job performance depending on the level of complexity (e.g., Hunter, Reference Hunter1986). They also note other potential aspects such as safety and satisfaction. The influence of job complexity has not been assessed nearly as often (or as well) in the employment interview literature. In a limited exception, Huffcutt, Conway, Roth, and Klehe (Reference Huffcutt, Conway, Roth and Klehe2004) found that the validity of one type of structured interview was affected by complexity while the validity for another type was not.
Of curiosity is the finding by Huffcutt, Roth, and McDaniel (Reference Huffcutt, Roth and McDaniel1996) that interviewer ratings tend to be more strongly associated with GMA for positions of lower complexity than for positions of higher complexity. We speculate that differences in GMA may tend to be greater among candidates for a worker position than among professional candidates (e.g., all candidates for an engineering position are likely to be reasonably bright) and that those differences exert greater influence on interviewee performance (see Kluemper, McLarty, Bishop, & Sen, Reference Kluemper, McLarty, Bishop and Sen2015).
We concur with Bergman and Jean (Reference Bergman and Jean2016) that worker samples tend to be underrepresented in organizational research, which could have deleterious effects on the conclusions and practices derived from this research. However, we argue that the effects of underrepresentation could vary considerably by organizational domain (e.g., selection vs. leadership vs. satisfaction). Our focus with this commentary is to address its effects on organizational selection, particularly with employment interviews, including issues such as criterion-related validity, prevalence of student samples, and the four criteria outlined by the focal authors (overlooked phenomena, differences in construct meaning, worker status, and human capital patterns).
To provide an empirical basis for this commentary, we pulled studies from our recent validity meta-analysis of employment interviews (Huffcutt, Culbertson, & Weyhrauch, Reference Huffcutt, Culbertson and Weyhrauch2014). In support of the validity analysis portion, we focused exclusively on the 73 studies with indirect range restriction only (i.e., the interview was not used to make selection decisions). This was the largest group, and doing so allowed us to analyze a group of studies with the same basic range restriction mechanism.
Underrepresentation of Worker Samples in Employment Interview Research
To provide an empirical assessment, we classified the position involved in each of our 73 studies as either “worker” or “professional.” The result was 27 worker samples, spanning occupations such as mill employees, clerical/administrative support, retail sales, electricians, day care workers, and telecommunication phone reps, and 41 professional samples spanning occupations such as entry-level managers, life insurance agents, Federal Bureau of Investigation agents, teachers, school principals, social workers, and medical school residents. We were unable to classify the remaining five studies, either because of a lack of information or because the occupation seemed to incorporate aspects of both classifications (e.g., special education paraprofessionals).
Clearly the worker samples were fewer in number than the professional samples in the 68 classified studies (27/68 or 39.7%). Bergman and Jean noted that over half of occupations are workers, thus confirming underrepresentation with this aspect of selection. However, the degree of underrepresentation was considerably less severe than in the general organizational studies sampled by the focal authors, among which they found only 9% worker samples. This difference could simply be a sampling artifact. Alternatively, it supports the notion that the degree of underrepresentation (and its potential effects) varies by organizational domain.
Use of Student Samples in Employment Interview Research
Whereas Bergman and Jean noted that 24% of their samples were students, only four of our 68 classified interview studies involved students (5.9%). Moreover, these four studies involved professional occupations, and the students were engaged in activities related to that profession. To illustrate, three of the four studies involved medical school residents performing rounds and treating patients.
The lower concentration of student samples in our interview dataset was most likely a result of the screening criteria in the Huffcutt et al. (Reference Huffcutt, Culbertson and Weyhrauch2014) meta-analysis. As noted, student samples were included only if they performed at least some of the regular duties of their profession, thereby excluding pure student samples (e.g., from a laboratory experiment).
This lower concentration highlights a potentially important point regarding primary versus meta-analytic research. Looking at primary research, it is likely that more student samples of various types (including laboratory) are included. However, meta-analytic syntheses tend to screen out the samples that don't have a direct tie to the intended occupation. Thus, we disagree somewhat with Bergman and Jean's notion that “I-O psychology's penchant for meta-analysis actually compounds many of the problems discussed herein” (p. 106). Instead, we argue that the inclusion criteria commonly associated with meta-analytic methodology may serve to mitigate some of the effects of worker underrepresentation.
Occupational Differences in Criterion-Related Validity
Research with general mental ability (GMA) has found that job complexity exerts a strong moderating influence on its validity. To illustrate, Hunter, Schmidt, and Le (Reference Hunter, Schmidt and Le2006) found that mean validity varied from .39 for the least complex job family to .73 for the highest. Although the worker versus professional distinction is not synonymous with job complexity, they should have a close association as worker occupations in general tend to be less complex than professional ones.
To test the influence of occupation, we computed the mean criterion-related validity across our 27 worker samples and our 41 professional samples. Because the focal article focused on occupational classifications and percentage representations therein (i.e., without reference to sample size), we computed a simple unweighted mean. The means were virtually identical, namely, .32 for workers and .33 for professionals. We note these values are uncorrected for range restriction, but given that all studies had indirect restriction only, the correction is unlikely to change the conclusion that they are highly similar.
However, structure tends to exert a strong influence on validity (see Huffcutt et al., Reference Huffcutt, Culbertson and Weyhrauch2014), and it is possible that the overall level of structure was different in the worker and professional samples. To assess this potential confound, we computed the average structure level within each classification using the Level 1 (no structure) to Level 4 (complete standardization) framework. The means were 3.6 for worker samples and 3.3 for professional samples, suggesting that worker validity might have been slightly overestimated relative to professional validity because the overall level of structure was slightly higher for the worker samples. Still, the magnitude should not have been extensive.
Thus, contrary to established findings with GMA, our analysis showed very little (if any) moderation of criterion-related validity from occupation (i.e., worker vs. professional) with interviews. In addition to varying across organizational domains, these findings suggest that the effects of worker underrepresentation may also vary within domain (e.g., interviews vs. GMA).
Bergman and Jean's Four Criteria
Now we turn to the four potential ways identified by the focal authors that worker sample underrepresentation could influence the results of organizational research. Our focus with this examination remains with the selection domain.
1. Overlooked Phenomena
There are two key reasons why potentially important phenomenon could be overlooked in selection of workers. First, given their higher prevalence (and sometimes importance), the operationalization of constructs in professional samples could carry over into worker samples even if the construct takes on different meaning in the latter. For instance, a job analyst highly attuned to developing selection systems for professional positions might also operationalize flexibility for worker positions more in terms of learning and knowledge even though variations in shift scheduling may be more crucial.
Second, there appears to be somewhat of a general tendency across occupations for job analysts to focus more strongly on knowledge, skills, and abilities (i.e., KSAs) than on the other (i.e., “O”) aspects of a position. A predominant focus on KSAs may suffice for professional positions where KSAs collectively tend to be more prevalent and important than other characteristics. But, for some (perhaps many) worker occupations, the other characteristics can dominate in terms of both occurrence and importance. For instance, schedule flexibility can be critical with some worker positions (e.g., fast food), and aspects such as willingness to stay late, cover additional shifts on short notice, and rotate assigned shifts (e.g., first vs. second) should all fall under “O.”
Continuing, one potential result of economic tenuousness, job insecurity, and holding second jobs could be less tenure. The process of replacing an employee is time consuming and costly even for the lowest level positions. Given the propensity for high turnover in a number of worker occupations (e.g., 100% or more is common in retail and fast food), it would seem logical for selection processes to include some elements designed to identify individuals who are more likely to stay a reasonable length of time with the organization.
To provide at least a partial assessment of whether flexibility and tenure are being overlooked in worker samples, we picked four among our database of 27 worker samples that were highly likely to involve schedule instability and have high turnover. They included clothing sales associates (Roth, Van Iddekinge, Huffcutt, Eidson, &Schmitt, Reference Roth, Van Iddekinge, Huffcutt, Eidson and Schmitt2005); home improvement store employees (Little, Schoenfelt, & Brown, Reference Little, Schoenfelt and Brown2000); appliance factor service technicians (Delery, Wright, McArthur, & Anderson, Reference Delery, Wright, McArthur and Anderson1994); and discount clothing, accessories, and home products sales associates (Motowidlo, Brownlee, & Schmit, Reference Motowidlo, Brownlee and Schmit2008).
Clothing sales associate. The main selection mechanism described in this study was a highly structured interview, although a Big Five personality test was also administered. There were six interview questions, designed respectively to assess (a) helping customers pick a gift, (b) helping customers match clothes, (c) helping when off the clock, (d) dealing with customers who ask for a discount, (e) identifying dishonest customers, and (f) dealing with an upset customer. The majority of these six content areas appear to be focused on the core KSA elements of the retail sales environment. Only one (helping when off the clock) appears to relate to schedule instability, although only to staying later and not variability in shift scheduling (e.g., start/end time) and/or covering additional shifts on short notice. None of the dimensions appear to address tenure, although that could have been covered via other selection mechanisms such as looking for “job-hopping” in the résumé.
Regarding the personality test, the primary purpose of its administration appears to have been assessing personality saturation in the interview questions. Regardless, to assess flexibility and tenure might have required a stronger focus on specific facets of the Big Five. For instance, impulsiveness might be the most relevant facet of neuroticism for tenure, while action might be the most relevant facet of openness for flexibility.
Home improvement store employees. A job analysis identified five core dimensions: (a) initiative, (b) problem solving, (c) communication/interpersonal skills, (d) quality, and (e) product knowledge. We note that schedule flexibility is not included as a dimension. Nor is there anything related to tenure, although again it might have been addressed elsewhere in the selection process.
Factory service technicians. A job analysis identified two main clusters, one for technical prowess and the other for interpersonal skills/customer service orientation. Both mental ability tests and a structured interview were included in their selection battery. Here there is a strong focus on mental/technical skills, which is not surprising given that appliances can malfunction in a variety of different ways, and the ability to diagnose problems and identify solutions quickly is a distinct advantage. The focus on customer interaction is also not surprising given that quick diagnoses and repair often cannot compensate for poor customer skills. What we do not see in this study is a reference to schedule flexibility, which we presume is common in this job area. For instance, the last job of the day may take considerably longer than anticipated, thereby requiring the technician to stay later than the target quitting time.
Discount clothing, accessories, and home products sales associates. Here the job analysis itself focused directly on customer interaction. From this analysis, a selection battery containing an intelligence test, a personality test, several roleplays, and a structured interview was developed. Selection content oriented toward the workers themselves (e.g., flexibility, tenure) appears to be absent.
In summary, these four studies suggest the possibility that at least some phenomenon unique to (or at least more strongly associated with) worker occupations could be overlooked. Only one of these four studies included flexibility in scheduling as a dimension, and even then, it was limited. A common element across these studies was the use of a critical incident job analysis. It is possible that the individuals contributing critical incidents tend to think of issues pertaining directly to the work itself (e.g., knowledge/learning, customer contact) rather than tangential issues such as scheduling and reducing turnover. Alternatively, it is possible that at least some critical incidents related to more tangential issues emerge, but the job analysts choose to focus more on those pertaining directly to core work issues (i.e., the KSAs).
2. Differences in Construct Meaning
Referencing the four worker samples just discussed in relation to overlooked phenomenon, we see no evidence for differences in construct meaning. For instance, the one study that did include flexibility as a dimension interpreted it as pertaining to scheduling rather than to the work itself. We believe it is the job analysis process that forms the core of selection system design that helps keep constructs operationalized correctly for both worker and professional samples, whether that analysis is based on tasks or critical incidents.
3. Worker Status
As evidence for differences in worker status, Bergman and Jean noted that GMA has been found to differentially predict job performance depending on the level of complexity (e.g., Hunter, Reference Hunter1986). They also note other potential aspects such as safety and satisfaction. The influence of job complexity has not been assessed nearly as often (or as well) in the employment interview literature. In a limited exception, Huffcutt, Conway, Roth, and Klehe (Reference Huffcutt, Conway, Roth and Klehe2004) found that the validity of one type of structured interview was affected by complexity while the validity for another type was not.
Of curiosity is the finding by Huffcutt, Roth, and McDaniel (Reference Huffcutt, Roth and McDaniel1996) that interviewer ratings tend to be more strongly associated with GMA for positions of lower complexity than for positions of higher complexity. We speculate that differences in GMA may tend to be greater among candidates for a worker position than among professional candidates (e.g., all candidates for an engineering position are likely to be reasonably bright) and that those differences exert greater influence on interviewee performance (see Kluemper, McLarty, Bishop, & Sen, Reference Kluemper, McLarty, Bishop and Sen2015).
4. Human Capital Patterns
Bergman and Jean note differences in human capital patterns such as the obtainment of more education and experiencing of greater overall health with professional positions (relative to workers). Regarding education, we believe this aspect has less relevance to selection given that minimal educational and/or training requirements are often used for preliminary screening and then not as strongly considered thereafter.
Assessment of potential health and longevity, however, should be an important consideration in selection. Nevertheless, such aspects are rarely included in the selection process other than being used as a preliminary screening mechanism. For instance, a job posting for a convenience store clerk position might note that candidates must be able to lift 15 pounds. Yet, lifting 15 pounds once as part of the selection process is very different from the capability to lift such weight repeatedly during a workday.
Parting Thoughts
Bergman and Jean put forth the proposition that worker samples tend to be underrepresented in organizational research and that such underrepresentation could have deleterious effects on conclusions and practices. Looking specifically at the organizational selection domain, we were able to confirm worker underrepresentation along with the possibility that some potentially important worker phenomena are being overlooked (particularly with “other” job aspects such as schedule flexibility and tenure).
However, the degree of worker underrepresentation does not appear to be nearly as severe in selection, nor was there any real evidence for differences in criterion-related validity, the prevalence of and/or influence from student samples, differences in construct meaning, or human capital patterns. A key reason for these differences appears to be the screening out of less appropriate (or relevant) studies from the screening criteria associated with the meta-analytic process.
In terms of application, we believe researchers and practitioners may need to “cast a wider net” during the job analysis process for worker occupations. Specifically, job analysts can strive to ensure that “O” components such as schedule flexibility, safety, and possibly tenure are given adequate attention and represented fully in the selection mechanisms that are developed (e.g., a structured interview).