Bergman and Jean (Reference Bergman and Jean2016) compare published industrial and organizational (I-O) literature with labor statistics, demonstrating an underrepresentation of “workers” (i.e., “wage earners, laborers, first-line personnel, freelancers, contract workers”) relative to managerial, professional, and executive positions. They note that one of four ways in which worker underrepresentation undermines the utility of I-O psychology research is that we could miss the role of worker status as a main effect on important variables and/or a moderator of key relationships, which could hinder understanding of important phenomena as they relate to workers. We applaud the emphasis on workers and agree with this basic premise.
In order to draw valid inferences regarding the strength of an attitude or prevalence of an opinion—that is, generalizing a sample statistic to a larger population—workers indeed ought to be sampled if they are part of the population to which one is trying to generalize. However, when trying to generalize inferences about behavior, as in theory testing, the particular sample used might not be a critical concern unless (a) the research contains a specific and well-defined population of interest, (b) there are important differences in the motivation of respondents, and/or (c) there is reason to believe that features of the sample have a systematic influence on the relationships of interest (Highhouse & Gillespie, Reference Highhouse, Gillespie, Lance and Vandenberg2009, pp. 261–262).
Therefore, for any given study, we might ask, Is there compelling rationale or evidence to suggest that worker status has a systematic influence on key variables or relationships? Assuming the field is unlikely to reverse publication trends enough to represent workers proportionally, more research is needed that deliberately tests (or theorizes) the extent to which descriptions, theories, patterns of findings, and effects apply to “workers” as compared with “nonworkers.” In other words, we suggest that the importance of sample composition depends on the research question, and it may be possible in some instances to test empirically the extent to which descriptions, theories, patterns of findings, and effects apply to “workers” as compared with “nonworkers.” As an illustration, we use survey data representative of the United States working population to test for main effects of worker status on job satisfaction and moderated effects of worker status on the relationship between job satisfaction and outcomes.
Method
Our data were originally collected to update, renorm, and provide validity evidence for the Job Descriptive Index (JDI), a faceted job satisfaction measure; Smith, Kendall, & Hulin, Reference Smith, Kendall and Hulin1969) and the Job in General (JIG, an overall job satisfaction measure; Ironson, Smith, Brannick, Gibson, & Paul, Reference Ironson, Smith, Brannick, Gibson and Paul1989; see Gillespie et al., in press). Participants (N = 1,400) were drawn from an online marketing research panel using stratified random sampling with subgroup quotas to represent the U.S. working population on major demographic and industry variables. The survey included basic demographic questions and a battery of measures, including the JDI, JIG, Stress in General (SIG; Yankelevich, Broadfoot, Gillespie, Gillespie, & Guidroz, Reference Yankelevich, Broadfoot, Gillespie, Gillespie and Guidroz2012), Intentions to Quit (ITQ; Crossley, Grauer, Lin, & Stanton, Reference Crossley, Grauer, Lin and Stanton2002), and Trustworthiness of Management (TOM; see Parra, Reference Parra1995). The sample represented the U.S. working population with respect to industry and closely approximated the population in terms of age and gender. “Manager status” was used as the closest available proxy for “worker status.” Thirty-two percent (n = 449) of the sample classified as a manager, and 68% (n = 951) classified as nonmanager.
Results
First, a series of independent-samples t tests were conducted to test for the main effects of manager status on job attitudes. Then, multiple regression analyses were run to test whether manager status moderated the relationship between job satisfaction and ITQ, TOM, and SIG. Using G*Power's sensitivity analysis, the sample provided 95% power to detect small effect sizes: a Cohen's d of .21 for the main effect t tests (two-tailed) and a ΔR
2 of .015 for the moderated regressions (α = .05; Faul, Erdfelder, Buchner, & Lang, Reference Faul, Erdfelder, Buchner and Lang2009).
Significant differences were observed between “workers” (i.e., nonmanagers) and “nonworkers” (i.e., managers) on the JDI facets of satisfaction with work, pay, and promotion opportunities, and on the JIG (ps < .001), with nonworkers reporting greater satisfaction than workers. No significant differences were found on the JDI facets of satisfaction with supervision or coworkers (ps ≈ .30).
To test the role of worker status as a moderator, SIG, ITQ, and TOM were regressed separately onto (a) all of the JDI facets, the JIG, and managerial status (Step 1) and (b) interaction terms comprising worker status by JDI facets and JIG (Step 2). Results for Step 2 of these three regressions were all nonsignificant (all p-values ≥ .13), indicating that worker status was not found to moderate the relationship between the job satisfaction variables and criteria.
Discussion
We found that worker status (operationalized as manager status here) was related to job satisfaction. This main effect is consequential because descriptive inferences based on point estimates (e.g., job satisfaction scores) are used for decision making (Guion, Reference Guion2011). A practical implication is decision makers need to pay attention to employee type when drawing inferences about levels of job satisfaction (i.e., generalizing sample statistics to larger populations), which is why subgroup norms are made available for the JDI and JIG based on worker (i.e., manager) status and other demographics (Gillespie et al., in press). The research implication of this is that a thorough treatment of job satisfaction ought to include worker status as a determinant (Robie, Ryan, Schmieder, Parra, & Smith, Reference Robie, Ryan, Schmieder, Parra and Smith1998).
By contrast, we did not find a moderated effect of worker status on the relationship between job satisfaction and other variables in the nomological network that we have used for criterion-related validity evidence (i.e., SIG, ITQ, TOM). Because we drew from a sample designed to represent the full-time U.S. workforce and had 95% power to detect a small interaction effect, we can be reasonably confident that such an effect does not exist in the population. We had no a priori reason to expect such differences. Rather, we simply provided an empirical test to ascertain whether theoretical inferences about these constructs, when obtained from “nonworker” samples (e.g., managers, students), might still be useful for worker populations as well (Bergman & Jean).
Together, these two sets of findings bear out our initial claim that (a) sample composition is critical when generalizing a sample statistic to a larger population, yet (b) representativeness may not always be a concern for theory testing. At face value, our nonsignificant results for worker status as a moderator belie Bergman and Jean's main argument. However, it nevertheless demonstrates that researchers may choose to take their concern seriously by conducting empirical tests to determine whether or not worker status moderates important relationships, providing guidance about domains in which worker representation is most critical. There is a limitation of our position though: When phenomena are disproportionally experienced by an underrepresented worker population (e.g., food insufficiency), the field will still overlook such phenomena unless workers are included in the research. In addition, there are surely other theoretical inferences for which worker status is a more plausible moderator, and research identifying such instances would be beneficial.
In sum, the importance of sample composition depends on the research question, and the impact of sample type can under some conditions be tested empirically. It follows that workers may not actually need to be proportionally represented in all research; rather, it may be sufficient to test the conditions under which worker status is a factor on decisions, findings, and theories. Thus, in some instances, it may be possible for researchers to make theoretical inferences about phenomena related to workers, even if the sample includes people other than workers (e.g., managers, students). An important implication of this point is that, in order to demonstrate instances when worker representation does not seem to impact findings, nonsignificant results must be published. Testing the conditions under which worker status informs key findings helps ensure workers get a fair, if not proportional, treatment in the literature, thereby increasing the utility of I-O research for science and practice.
Bergman and Jean (Reference Bergman and Jean2016) compare published industrial and organizational (I-O) literature with labor statistics, demonstrating an underrepresentation of “workers” (i.e., “wage earners, laborers, first-line personnel, freelancers, contract workers”) relative to managerial, professional, and executive positions. They note that one of four ways in which worker underrepresentation undermines the utility of I-O psychology research is that we could miss the role of worker status as a main effect on important variables and/or a moderator of key relationships, which could hinder understanding of important phenomena as they relate to workers. We applaud the emphasis on workers and agree with this basic premise.
In order to draw valid inferences regarding the strength of an attitude or prevalence of an opinion—that is, generalizing a sample statistic to a larger population—workers indeed ought to be sampled if they are part of the population to which one is trying to generalize. However, when trying to generalize inferences about behavior, as in theory testing, the particular sample used might not be a critical concern unless (a) the research contains a specific and well-defined population of interest, (b) there are important differences in the motivation of respondents, and/or (c) there is reason to believe that features of the sample have a systematic influence on the relationships of interest (Highhouse & Gillespie, Reference Highhouse, Gillespie, Lance and Vandenberg2009, pp. 261–262).
Therefore, for any given study, we might ask, Is there compelling rationale or evidence to suggest that worker status has a systematic influence on key variables or relationships? Assuming the field is unlikely to reverse publication trends enough to represent workers proportionally, more research is needed that deliberately tests (or theorizes) the extent to which descriptions, theories, patterns of findings, and effects apply to “workers” as compared with “nonworkers.” In other words, we suggest that the importance of sample composition depends on the research question, and it may be possible in some instances to test empirically the extent to which descriptions, theories, patterns of findings, and effects apply to “workers” as compared with “nonworkers.” As an illustration, we use survey data representative of the United States working population to test for main effects of worker status on job satisfaction and moderated effects of worker status on the relationship between job satisfaction and outcomes.
Method
Our data were originally collected to update, renorm, and provide validity evidence for the Job Descriptive Index (JDI), a faceted job satisfaction measure; Smith, Kendall, & Hulin, Reference Smith, Kendall and Hulin1969) and the Job in General (JIG, an overall job satisfaction measure; Ironson, Smith, Brannick, Gibson, & Paul, Reference Ironson, Smith, Brannick, Gibson and Paul1989; see Gillespie et al., in press). Participants (N = 1,400) were drawn from an online marketing research panel using stratified random sampling with subgroup quotas to represent the U.S. working population on major demographic and industry variables. The survey included basic demographic questions and a battery of measures, including the JDI, JIG, Stress in General (SIG; Yankelevich, Broadfoot, Gillespie, Gillespie, & Guidroz, Reference Yankelevich, Broadfoot, Gillespie, Gillespie and Guidroz2012), Intentions to Quit (ITQ; Crossley, Grauer, Lin, & Stanton, Reference Crossley, Grauer, Lin and Stanton2002), and Trustworthiness of Management (TOM; see Parra, Reference Parra1995). The sample represented the U.S. working population with respect to industry and closely approximated the population in terms of age and gender. “Manager status” was used as the closest available proxy for “worker status.” Thirty-two percent (n = 449) of the sample classified as a manager, and 68% (n = 951) classified as nonmanager.
Results
First, a series of independent-samples t tests were conducted to test for the main effects of manager status on job attitudes. Then, multiple regression analyses were run to test whether manager status moderated the relationship between job satisfaction and ITQ, TOM, and SIG. Using G*Power's sensitivity analysis, the sample provided 95% power to detect small effect sizes: a Cohen's d of .21 for the main effect t tests (two-tailed) and a ΔR 2 of .015 for the moderated regressions (α = .05; Faul, Erdfelder, Buchner, & Lang, Reference Faul, Erdfelder, Buchner and Lang2009).
Significant differences were observed between “workers” (i.e., nonmanagers) and “nonworkers” (i.e., managers) on the JDI facets of satisfaction with work, pay, and promotion opportunities, and on the JIG (ps < .001), with nonworkers reporting greater satisfaction than workers. No significant differences were found on the JDI facets of satisfaction with supervision or coworkers (ps ≈ .30).
To test the role of worker status as a moderator, SIG, ITQ, and TOM were regressed separately onto (a) all of the JDI facets, the JIG, and managerial status (Step 1) and (b) interaction terms comprising worker status by JDI facets and JIG (Step 2). Results for Step 2 of these three regressions were all nonsignificant (all p-values ≥ .13), indicating that worker status was not found to moderate the relationship between the job satisfaction variables and criteria.
Discussion
We found that worker status (operationalized as manager status here) was related to job satisfaction. This main effect is consequential because descriptive inferences based on point estimates (e.g., job satisfaction scores) are used for decision making (Guion, Reference Guion2011). A practical implication is decision makers need to pay attention to employee type when drawing inferences about levels of job satisfaction (i.e., generalizing sample statistics to larger populations), which is why subgroup norms are made available for the JDI and JIG based on worker (i.e., manager) status and other demographics (Gillespie et al., in press). The research implication of this is that a thorough treatment of job satisfaction ought to include worker status as a determinant (Robie, Ryan, Schmieder, Parra, & Smith, Reference Robie, Ryan, Schmieder, Parra and Smith1998).
By contrast, we did not find a moderated effect of worker status on the relationship between job satisfaction and other variables in the nomological network that we have used for criterion-related validity evidence (i.e., SIG, ITQ, TOM). Because we drew from a sample designed to represent the full-time U.S. workforce and had 95% power to detect a small interaction effect, we can be reasonably confident that such an effect does not exist in the population. We had no a priori reason to expect such differences. Rather, we simply provided an empirical test to ascertain whether theoretical inferences about these constructs, when obtained from “nonworker” samples (e.g., managers, students), might still be useful for worker populations as well (Bergman & Jean).
Together, these two sets of findings bear out our initial claim that (a) sample composition is critical when generalizing a sample statistic to a larger population, yet (b) representativeness may not always be a concern for theory testing. At face value, our nonsignificant results for worker status as a moderator belie Bergman and Jean's main argument. However, it nevertheless demonstrates that researchers may choose to take their concern seriously by conducting empirical tests to determine whether or not worker status moderates important relationships, providing guidance about domains in which worker representation is most critical. There is a limitation of our position though: When phenomena are disproportionally experienced by an underrepresented worker population (e.g., food insufficiency), the field will still overlook such phenomena unless workers are included in the research. In addition, there are surely other theoretical inferences for which worker status is a more plausible moderator, and research identifying such instances would be beneficial.
In sum, the importance of sample composition depends on the research question, and the impact of sample type can under some conditions be tested empirically. It follows that workers may not actually need to be proportionally represented in all research; rather, it may be sufficient to test the conditions under which worker status is a factor on decisions, findings, and theories. Thus, in some instances, it may be possible for researchers to make theoretical inferences about phenomena related to workers, even if the sample includes people other than workers (e.g., managers, students). An important implication of this point is that, in order to demonstrate instances when worker representation does not seem to impact findings, nonsignificant results must be published. Testing the conditions under which worker status informs key findings helps ensure workers get a fair, if not proportional, treatment in the literature, thereby increasing the utility of I-O research for science and practice.