Two recent focal articles in this journal have addressed issues related to sample selection and generalizability of results (Bergman & Jean, Reference Bergman and Jean2016; Landers & Behrend, Reference Landers and Behrend2015). If Bergman and Jean are correct, gone are the days of the Hawthorne studies in which research focused on the majority of the human workforce: the working class. Instead, researchers are allegedly two to three times as likely to exclusively sample managers as they are to exclusively sample workers. Assuming this is true, Bergman and Jean are correct to address why this occurs and how it may impact the field. However, there are two critical issues that must be considered alongside these questions: ongoing changes in how work is conducted and temporal trends in work. A consideration of these issues should yield additional insights that may supplement the recommendations made by Bergman and Jean.
Issue 1: A Shift in the Focus and Format of Work
There is no shortage of predictions about the future of work or the changing nature of work (Cooper, Reference Cooper1999; Davis-Blake & Broschak, Reference Davis-Blake and Broschak2009; Landry, Mahesh, & Hartman, Reference Landry, Mahesh and Hartman2005). Bergman and Jean address contract work (conceptualizing contractors as workers) and note that nontraditional work arrangements are underrepresented in our samples. However, there exist a number of other emerging groups of employees worth considering in the I-O literature. In recent decades, technological advancements and globalization have led to an increased prevalence of outsourcing (see Domberger, Jensen, & Stonecash, Reference Domberger, Jensen and Stonecash2002; Holcomb & Hitt, Reference Holcomb and Hitt2007). In addition, advances in communication technology and data transfer have enabled more employees to work remotely, increasing the popularity of full-time or part-time telework arrangements (Herschel & Andrews, Reference Herschel and Andrews1997; WorldatWork, 2011). Economic factors such as the recent recession also impacted the amount of entrepreneurial activity in the United States (Fairlie, Reference Fairlie2014). Some organizations refocused their efforts in an attempt to adopt entrepreneurial principles and hire employees with entrepreneurial traits (Dess, Lumpkin, & McGee, Reference Dess, Lumpkin and McGee1999). As the number of outsourced jobs, remote workers, and entrepreneurial employees increases, research on these workers should grow, as should research on organizations that employ these workers.
Although research may necessarily lag behind organizational trends, it is critical for researchers to remain knowledgeable about the changing nature of work. The aforementioned developments imply a plethora of research questions, many of which have not been addressed (or even considered) in our field. Do offshore employees behave similarly to centrally located employees? How does an increased reliance on outsourcing impact employee morale and perceived job security? If an organization recruits employees with entrepreneurial traits, how does this practice affect retention rates or the values and behaviors of these employees? Perhaps it is time for organizational theories to better address practical concerns regarding the interplay between strategic organizational human resource practices and employee perceptions and behaviors.
There is plentiful research on telework (see Gajendran & Harrison, Reference Gajendran and Harrison2007), including research on how nonteleworkers view their teleworking colleagues (Golden, Reference Golden2007). However, most of this research involves simple comparisons between employees who telework and those who do not. Despite the increasing prevalence of alternative work arrangements, teleworkers are unlikely to be represented alongside employees with traditional work arrangements unless a study is directly addressing telework. The same could be said of offshore employees and workers with other alternative work arrangements.
Construct representation must be addressed alongside the issue of sample representation. The focal article is correct in implying that model misspecification may result from sampling negligence (in the form of ignoring a potential moderator). However, a more important issue may arise in that constructs of critical importance are likely to differ as a function of employee group. For example, employees with remote work assignments may value autonomy more than employees with traditional work assignments do. Employees or organizations with an entrepreneurial focus may value creativity and innovation more than employees and organizations that look like the prototypical organization considered by I-O researchers (Brazeal & Herbert, Reference Brazeal and Herbert1999; DuBrin, Reference DuBrin1991). Although these constructs certainly exist in the I-O literature (e.g., Hackman & Oldham, Reference Hackman and Oldham1976), they may exert differential influence based on sample characteristics. In addition, by underrepresenting (or failing to include) certain types of employees in our samples, we risk undervaluing (or entirely missing) constructs critically important to organizations. As work evolves, research on work should change accordingly.
Issue 2: Temporal Trends
In addition to the workers potentially misrepresented or unrepresented by researchers, it is also important for I-O psychologists to consider how the employee population is changing with time. For example, the aging and retirement of the “baby boomer” generation has important implications for organizations (see Kanfer & Ackerman, Reference Kanfer and Ackerman2004). Differences between knowledge and skill sets among exiting boomers and entering millennials may play a role in organizational theory.
The effects of technological advancement should again be considered as I-O psychology considers the changing organizational landscape. Although early fears that workers would all be replaced by robots seem misinformed (see Whitney, Reference Whitney1986), there has been considerable proliferation of automation in menial or repetitive tasks in manufacturing. In addition, automated answering services have become increasingly common in recent decades. These developments disproportionately affect workers.
Finally, expansion of leadership theory may lead some organizations to adopt a culture in which every employee is expected to embody some characteristics of leadership (e.g., Birkinshaw, Reference Birkinshaw2014; Manz, Shipper, & Stewart, Reference Manz, Shipper and Stewart2009). Changes in the ratio of leaders to followers (or managers to workers) have implications for organizational and interpersonal perceptions and behaviors. If I-O psychology aims to accurately reflect organizational characteristics, this trend should be reflected in I-O theory and literature as well as sample representation.
Bergman and Jean's results suggest the questions of whether and when a sampling shift occurred. Only one of the five journals included in their analysis (Journal of Applied Psychology) existed prior to the Ohio State leadership studies in the mid-1940s. Perhaps the proliferation of leadership research in the 1940s to 1960s or the emergence of management as an academic discipline played a role in sampling preferences in industrial and organizational psychology. The analysis conducted in the focal article spans only 3 years (2012 to 2014) and is therefore unable to address the representativeness of I-O samples throughout the history of the field. Similarly, the scope of the analysis is incapable of determining the extent to which I-O samples have shifted to reflect the aforementioned temporal trends. Nor does the focal article address whether or when these trends should be reflected in sampling practices. Still, it is worth considering the extent to which researchers should be cognizant of these trends and actively attempt to mirror them in sampling practices.
Revisiting the Recommendations
Work is constantly changing, and a scientific field dedicated to studying work should address change as it occurs. The conceptualization of employees as workers or managers is too simplistic, as Bergman and Jean acknowledge. Nevertheless, the issues noted above may simultaneously affect workers, managers, and the proportion of workers to managers. Whether work has changed as a result of an aging workforce, the adoption of new technology, or a shift in the constructs of interest to practitioners, it is imperative that I-O researchers remain knowledgeable about these changes and reflect the organizational zeitgeist in research to the best of our ability. With this in mind, I would like to revisit the recommendations put forth in the focal article, noting some additional concerns.
Encourage Replication With Worker Samples
Few would argue with Bergman and Jean's implication that it is unwise to generalize results from a study of hourly manufacturing employees to upper-level management (or vice versa). Although the discovery of broad nomothetic relationships may be desirable, it is not always appropriate for a discussion section. Bergman and Jean are correct in asserting that replication is tacitly discouraged in many top journals, but the expansion of a theory or finding to a different population (e.g., workers) may not be considered a strict replication because the sample characteristics differ across studies. This is not to imply that journals would enthusiastically publish an otherwise carbon copy study in which the only novel characteristic is the sample. However, the expansion of a finding, relationship, or theory to a new population is routinely published in subfields such as cross-cultural research and scale development, provided the authors adequately delineate the contribution of this expansion to broader scientific knowledge.
Although the peer review and publication processes are not blameless in the misrepresentation of I-O samples, the onus of good science ultimately lies with the investigators. If a phenomenon applies to multiple employee groups, researchers should include each of those groups in the sample. Researchers who suspect that a relationship is potentially moderated by employee group should sample multiple groups and conduct moderation analyses. If a sample only reflects a single employee group, researchers should not generalize the results beyond that group.
Finally, the focal article is correct in asserting that meta-analysts should acknowledge the presence of sample homogeneity in primary source samples and interpret cumulative effects accordingly. Even one of validity generalization's greatest proponents knew the importance of empirically examining potential moderating effects such as job class/complexity (Hunter, Reference Hunter1980; Hunter & Hunter, Reference Hunter and Hunter1984).Footnote
1
If we suspect our theories to have boundary conditions such as moderation (see Bacharach, Reference Bacharach1989), then these conditions need to be empirically evaluated rather than willfully ignored in the service of convenience. Neither investigators nor meta-analysts should assume the generalization of validity without adequate evaluation of moderating effects, and one of the most likely moderating effects involves the type of work performed by an employee.
Sample Workers Using Different Strategies
Many of the suggestions in the focal article simply involve exchanging one problematic research practice with another. The authors note the problems associated with sampling from trade groups. However, similar problems pertaining to selection bias exist with Internet panel data and unions. Sampling based on property value is intriguing, but this practice may simply involve exchanging an undesirable form of sample homogeneity (i.e., employee type) for an undesirable form of sample heterogeneity (i.e., industry, company, type of job). However, in the interest of improving sample representativeness and associated generalizability, the consideration of alternative sampling methodologies such as those based on socioeconomic factors may represent an improvement over current practices.
Because the focal article briefly addressed online data collection (through Internet panels or crowdsourcing websites such as Amazon's Mechanical Turk), it is important to note that this form of sampling is fraught with problems (see Harms & DeSimone, Reference Harms and DeSimone2015). Crowdsourced samples tend to suffer from unrepresentativeness, though less so than student samples. In particular, crowdsourced sampling potentially oversamples women, educated individuals, and younger participants (see Paolacci, Chandler, & Ipeirotis, Reference Paolacci, Chandler and Ipeirotis2010). In short, online data collection techniques may undermine the entire effort of using them to improve the generalizability of current sampling techniques by simply exchanging one form of misrepresentation for another.
Be Open to Qualitative Work
Qualitative research can certainly be informative to I-O psychology (Lee, Mitchell, & Sablynski, Reference Lee, Mitchell and Sablynski1999), but it has traditionally been plagued by a lack of consensus regarding quality and best practices (Amis & Silk, Reference Amis and Silk2008; Cassell & Symon, Reference Cassell and Symon2011). Although it is certainly important to generate new research questions and identify new areas of inquiry, it is worth revisiting the earlier point regarding adequate construct representation in organizational science. If we believe that our theories and relationships are generalizable across both managers and workers, then Bergman and Jean demonstrate that we are doing a poor job of providing evidence to support this belief. If we believe that these theories and relationships differ across employee groups, then assessing moderation may be the answer, though this is not currently possible given the nature of most of our samples. If we believe that different theories and relationships apply, or that different (and potentially unidentified) variables affect organizational outcomes among workers, then qualitative inquiry may be necessary to demonstrate that this is the case.
As always, a more thorough investigation of organizational phenomena can be achieved by avoiding the sole reliance on qualitative research and instead conducting mixed methods research (Hanson, Creswell, Plano Clark, Petska, & Creswell, Reference Hanson, Creswell, Plano Clark, Petska and Creswell2005; Stange, Crabtree, & Miller, Reference Stange, Crabtree and Miller2006). In doing so, the investigator can combine the benefits of exploratory qualitative inquiry with the rigor of confirmatory quantitative testing. This would allow researchers to discover new phenomena relevant to the worker population but also to quantitatively justify the importance of these discoveries to readers.
Conclusions
Combining the assertions made by Bergman and Jean with those presented in the earlier focal article by Landers and Behrend (Reference Landers and Behrend2015), it would seem that the state of sampling in I-O research is abysmal. To what extent do researchers consider generalizability prior to conducting a study and to what extent do we simply use what is available, ignoring external validity in favor of convenience? If researchers value convenience more than representativeness and misrepresent their populations of interest, then the entire merit of our field is questionable. If our theories apply to the entire spectrum of employees, then there is no valid reason to disproportionately favor manager samples over worker samples. In doing so, we are undermining our ability to conduct applied research that will be beneficial to organizations and their employees.
Organizational researchers have a duty to remain knowledgeable about changes in the workplace, whether these changes are caused by demographic shifts, technological advances, or economic or industry trends. These changes may necessitate reconsidering established theories through the introduction or exploration of emerging constructs of interest to organizations. New (and existing) relationships may be moderated by employee group, and this moderation can only be explored (in primary studies or meta-analyses) when adequate and representative sampling procedures exist. If we fail to adequately sample from the population of interest, then we risk misunderstanding and misconstruing the importance of every construct we study.
Ultimately it is simply a matter of conducting high-quality research and being a good consumer of research. Investigators should always consider external validity prior to conducting a study. It is never advisable to “settle” for a suboptimal sample, as this can significantly diminish or even negate the utility of one's investigation. It is advisable to address the appropriateness of one's sample in the introduction section or method section. Editors, reviewers, and readers should critically consider whether the sampling procedure is congruent with the rationale provided and the discussion offered by authors. Inadequate sampling and overgeneralization should never be considered an acceptable practice, much less the status quo in I-O psychology.
You have
Access
Two recent focal articles in this journal have addressed issues related to sample selection and generalizability of results (Bergman & Jean, Reference Bergman and Jean2016; Landers & Behrend, Reference Landers and Behrend2015). If Bergman and Jean are correct, gone are the days of the Hawthorne studies in which research focused on the majority of the human workforce: the working class. Instead, researchers are allegedly two to three times as likely to exclusively sample managers as they are to exclusively sample workers. Assuming this is true, Bergman and Jean are correct to address why this occurs and how it may impact the field. However, there are two critical issues that must be considered alongside these questions: ongoing changes in how work is conducted and temporal trends in work. A consideration of these issues should yield additional insights that may supplement the recommendations made by Bergman and Jean.
Issue 1: A Shift in the Focus and Format of Work
There is no shortage of predictions about the future of work or the changing nature of work (Cooper, Reference Cooper1999; Davis-Blake & Broschak, Reference Davis-Blake and Broschak2009; Landry, Mahesh, & Hartman, Reference Landry, Mahesh and Hartman2005). Bergman and Jean address contract work (conceptualizing contractors as workers) and note that nontraditional work arrangements are underrepresented in our samples. However, there exist a number of other emerging groups of employees worth considering in the I-O literature. In recent decades, technological advancements and globalization have led to an increased prevalence of outsourcing (see Domberger, Jensen, & Stonecash, Reference Domberger, Jensen and Stonecash2002; Holcomb & Hitt, Reference Holcomb and Hitt2007). In addition, advances in communication technology and data transfer have enabled more employees to work remotely, increasing the popularity of full-time or part-time telework arrangements (Herschel & Andrews, Reference Herschel and Andrews1997; WorldatWork, 2011). Economic factors such as the recent recession also impacted the amount of entrepreneurial activity in the United States (Fairlie, Reference Fairlie2014). Some organizations refocused their efforts in an attempt to adopt entrepreneurial principles and hire employees with entrepreneurial traits (Dess, Lumpkin, & McGee, Reference Dess, Lumpkin and McGee1999). As the number of outsourced jobs, remote workers, and entrepreneurial employees increases, research on these workers should grow, as should research on organizations that employ these workers.
Although research may necessarily lag behind organizational trends, it is critical for researchers to remain knowledgeable about the changing nature of work. The aforementioned developments imply a plethora of research questions, many of which have not been addressed (or even considered) in our field. Do offshore employees behave similarly to centrally located employees? How does an increased reliance on outsourcing impact employee morale and perceived job security? If an organization recruits employees with entrepreneurial traits, how does this practice affect retention rates or the values and behaviors of these employees? Perhaps it is time for organizational theories to better address practical concerns regarding the interplay between strategic organizational human resource practices and employee perceptions and behaviors.
There is plentiful research on telework (see Gajendran & Harrison, Reference Gajendran and Harrison2007), including research on how nonteleworkers view their teleworking colleagues (Golden, Reference Golden2007). However, most of this research involves simple comparisons between employees who telework and those who do not. Despite the increasing prevalence of alternative work arrangements, teleworkers are unlikely to be represented alongside employees with traditional work arrangements unless a study is directly addressing telework. The same could be said of offshore employees and workers with other alternative work arrangements.
Construct representation must be addressed alongside the issue of sample representation. The focal article is correct in implying that model misspecification may result from sampling negligence (in the form of ignoring a potential moderator). However, a more important issue may arise in that constructs of critical importance are likely to differ as a function of employee group. For example, employees with remote work assignments may value autonomy more than employees with traditional work assignments do. Employees or organizations with an entrepreneurial focus may value creativity and innovation more than employees and organizations that look like the prototypical organization considered by I-O researchers (Brazeal & Herbert, Reference Brazeal and Herbert1999; DuBrin, Reference DuBrin1991). Although these constructs certainly exist in the I-O literature (e.g., Hackman & Oldham, Reference Hackman and Oldham1976), they may exert differential influence based on sample characteristics. In addition, by underrepresenting (or failing to include) certain types of employees in our samples, we risk undervaluing (or entirely missing) constructs critically important to organizations. As work evolves, research on work should change accordingly.
Issue 2: Temporal Trends
In addition to the workers potentially misrepresented or unrepresented by researchers, it is also important for I-O psychologists to consider how the employee population is changing with time. For example, the aging and retirement of the “baby boomer” generation has important implications for organizations (see Kanfer & Ackerman, Reference Kanfer and Ackerman2004). Differences between knowledge and skill sets among exiting boomers and entering millennials may play a role in organizational theory.
The effects of technological advancement should again be considered as I-O psychology considers the changing organizational landscape. Although early fears that workers would all be replaced by robots seem misinformed (see Whitney, Reference Whitney1986), there has been considerable proliferation of automation in menial or repetitive tasks in manufacturing. In addition, automated answering services have become increasingly common in recent decades. These developments disproportionately affect workers.
Finally, expansion of leadership theory may lead some organizations to adopt a culture in which every employee is expected to embody some characteristics of leadership (e.g., Birkinshaw, Reference Birkinshaw2014; Manz, Shipper, & Stewart, Reference Manz, Shipper and Stewart2009). Changes in the ratio of leaders to followers (or managers to workers) have implications for organizational and interpersonal perceptions and behaviors. If I-O psychology aims to accurately reflect organizational characteristics, this trend should be reflected in I-O theory and literature as well as sample representation.
Bergman and Jean's results suggest the questions of whether and when a sampling shift occurred. Only one of the five journals included in their analysis (Journal of Applied Psychology) existed prior to the Ohio State leadership studies in the mid-1940s. Perhaps the proliferation of leadership research in the 1940s to 1960s or the emergence of management as an academic discipline played a role in sampling preferences in industrial and organizational psychology. The analysis conducted in the focal article spans only 3 years (2012 to 2014) and is therefore unable to address the representativeness of I-O samples throughout the history of the field. Similarly, the scope of the analysis is incapable of determining the extent to which I-O samples have shifted to reflect the aforementioned temporal trends. Nor does the focal article address whether or when these trends should be reflected in sampling practices. Still, it is worth considering the extent to which researchers should be cognizant of these trends and actively attempt to mirror them in sampling practices.
Revisiting the Recommendations
Work is constantly changing, and a scientific field dedicated to studying work should address change as it occurs. The conceptualization of employees as workers or managers is too simplistic, as Bergman and Jean acknowledge. Nevertheless, the issues noted above may simultaneously affect workers, managers, and the proportion of workers to managers. Whether work has changed as a result of an aging workforce, the adoption of new technology, or a shift in the constructs of interest to practitioners, it is imperative that I-O researchers remain knowledgeable about these changes and reflect the organizational zeitgeist in research to the best of our ability. With this in mind, I would like to revisit the recommendations put forth in the focal article, noting some additional concerns.
Encourage Replication With Worker Samples
Few would argue with Bergman and Jean's implication that it is unwise to generalize results from a study of hourly manufacturing employees to upper-level management (or vice versa). Although the discovery of broad nomothetic relationships may be desirable, it is not always appropriate for a discussion section. Bergman and Jean are correct in asserting that replication is tacitly discouraged in many top journals, but the expansion of a theory or finding to a different population (e.g., workers) may not be considered a strict replication because the sample characteristics differ across studies. This is not to imply that journals would enthusiastically publish an otherwise carbon copy study in which the only novel characteristic is the sample. However, the expansion of a finding, relationship, or theory to a new population is routinely published in subfields such as cross-cultural research and scale development, provided the authors adequately delineate the contribution of this expansion to broader scientific knowledge.
Although the peer review and publication processes are not blameless in the misrepresentation of I-O samples, the onus of good science ultimately lies with the investigators. If a phenomenon applies to multiple employee groups, researchers should include each of those groups in the sample. Researchers who suspect that a relationship is potentially moderated by employee group should sample multiple groups and conduct moderation analyses. If a sample only reflects a single employee group, researchers should not generalize the results beyond that group.
Finally, the focal article is correct in asserting that meta-analysts should acknowledge the presence of sample homogeneity in primary source samples and interpret cumulative effects accordingly. Even one of validity generalization's greatest proponents knew the importance of empirically examining potential moderating effects such as job class/complexity (Hunter, Reference Hunter1980; Hunter & Hunter, Reference Hunter and Hunter1984).Footnote 1 If we suspect our theories to have boundary conditions such as moderation (see Bacharach, Reference Bacharach1989), then these conditions need to be empirically evaluated rather than willfully ignored in the service of convenience. Neither investigators nor meta-analysts should assume the generalization of validity without adequate evaluation of moderating effects, and one of the most likely moderating effects involves the type of work performed by an employee.
Sample Workers Using Different Strategies
Many of the suggestions in the focal article simply involve exchanging one problematic research practice with another. The authors note the problems associated with sampling from trade groups. However, similar problems pertaining to selection bias exist with Internet panel data and unions. Sampling based on property value is intriguing, but this practice may simply involve exchanging an undesirable form of sample homogeneity (i.e., employee type) for an undesirable form of sample heterogeneity (i.e., industry, company, type of job). However, in the interest of improving sample representativeness and associated generalizability, the consideration of alternative sampling methodologies such as those based on socioeconomic factors may represent an improvement over current practices.
Because the focal article briefly addressed online data collection (through Internet panels or crowdsourcing websites such as Amazon's Mechanical Turk), it is important to note that this form of sampling is fraught with problems (see Harms & DeSimone, Reference Harms and DeSimone2015). Crowdsourced samples tend to suffer from unrepresentativeness, though less so than student samples. In particular, crowdsourced sampling potentially oversamples women, educated individuals, and younger participants (see Paolacci, Chandler, & Ipeirotis, Reference Paolacci, Chandler and Ipeirotis2010). In short, online data collection techniques may undermine the entire effort of using them to improve the generalizability of current sampling techniques by simply exchanging one form of misrepresentation for another.
Be Open to Qualitative Work
Qualitative research can certainly be informative to I-O psychology (Lee, Mitchell, & Sablynski, Reference Lee, Mitchell and Sablynski1999), but it has traditionally been plagued by a lack of consensus regarding quality and best practices (Amis & Silk, Reference Amis and Silk2008; Cassell & Symon, Reference Cassell and Symon2011). Although it is certainly important to generate new research questions and identify new areas of inquiry, it is worth revisiting the earlier point regarding adequate construct representation in organizational science. If we believe that our theories and relationships are generalizable across both managers and workers, then Bergman and Jean demonstrate that we are doing a poor job of providing evidence to support this belief. If we believe that these theories and relationships differ across employee groups, then assessing moderation may be the answer, though this is not currently possible given the nature of most of our samples. If we believe that different theories and relationships apply, or that different (and potentially unidentified) variables affect organizational outcomes among workers, then qualitative inquiry may be necessary to demonstrate that this is the case.
As always, a more thorough investigation of organizational phenomena can be achieved by avoiding the sole reliance on qualitative research and instead conducting mixed methods research (Hanson, Creswell, Plano Clark, Petska, & Creswell, Reference Hanson, Creswell, Plano Clark, Petska and Creswell2005; Stange, Crabtree, & Miller, Reference Stange, Crabtree and Miller2006). In doing so, the investigator can combine the benefits of exploratory qualitative inquiry with the rigor of confirmatory quantitative testing. This would allow researchers to discover new phenomena relevant to the worker population but also to quantitatively justify the importance of these discoveries to readers.
Conclusions
Combining the assertions made by Bergman and Jean with those presented in the earlier focal article by Landers and Behrend (Reference Landers and Behrend2015), it would seem that the state of sampling in I-O research is abysmal. To what extent do researchers consider generalizability prior to conducting a study and to what extent do we simply use what is available, ignoring external validity in favor of convenience? If researchers value convenience more than representativeness and misrepresent their populations of interest, then the entire merit of our field is questionable. If our theories apply to the entire spectrum of employees, then there is no valid reason to disproportionately favor manager samples over worker samples. In doing so, we are undermining our ability to conduct applied research that will be beneficial to organizations and their employees.
Organizational researchers have a duty to remain knowledgeable about changes in the workplace, whether these changes are caused by demographic shifts, technological advances, or economic or industry trends. These changes may necessitate reconsidering established theories through the introduction or exploration of emerging constructs of interest to organizations. New (and existing) relationships may be moderated by employee group, and this moderation can only be explored (in primary studies or meta-analyses) when adequate and representative sampling procedures exist. If we fail to adequately sample from the population of interest, then we risk misunderstanding and misconstruing the importance of every construct we study.
Ultimately it is simply a matter of conducting high-quality research and being a good consumer of research. Investigators should always consider external validity prior to conducting a study. It is never advisable to “settle” for a suboptimal sample, as this can significantly diminish or even negate the utility of one's investigation. It is advisable to address the appropriateness of one's sample in the introduction section or method section. Editors, reviewers, and readers should critically consider whether the sampling procedure is congruent with the rationale provided and the discussion offered by authors. Inadequate sampling and overgeneralization should never be considered an acceptable practice, much less the status quo in I-O psychology.