Schneider and Pulakos (Reference Schneider and Pulakos2022) make a strong case for industrial-organizational (I-O) research to expand from a focus on individual differences to the study of organizational differences or what they refer to as an organizational mindset. They provide a number of examples of organizational level research but they do not discuss what is probably the most extensively studied traditional I-O topic at the organizational level: training. In fact, there are so many organizational-level studies on training and organizational performance that two meta-analyses have been conducted.
Research on training and organizational performance
There have been over 200 published studies in 36 journals in the last 40 years on the relationship between training and organizational performance (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019). The first meta-analysis by Tharenou etal. (Reference Tharenou, Saks and Moore2007) included 67 studies and found that training is positively related to human resource outcomes and organizational performance and, to a lesser extent, financial outcomes. Amore recent meta-analysis by Garavan etal. (Reference Garavan, McCarthy, Lai, Murphy, Sheehan and Carbery2021) included 119 primary studies (313 effect sizes) published between 1985 and 2019. They found a significant and direct positive relationship between training and organizational performance. Both meta-analyses found similar effect sizes between training and organizational performance (.25 for Garavan etal., .21 for Tharenou etal., and .24 with perceptual measures of organizational performance). According to Garavan etal. (Reference Garavan, McCarthy, Lai, Murphy, Sheehan and Carbery2021), a 1-standard-deviation increase in training is associated with a .25-standard-deviation increase in organizational performance.
Given that research on training and organizational performance is the most studied traditional I-O topic at the organizational level, it is worth taking a closer look at this research to learn more about doing I-O research with an organizational mindset.
Lessons to learn from research on training and organizational performance
Research on training and organizational performance suggests that doing I-O research with an organizational mindset requires careful attention to and consideration of the independent variable, dependent variables, mediating and moderating variables, micro–macro integration, and methodological issues.
Independent variable
The independent variable of studies at the organizational level can be operationalized in many ways. Research on training and organizational performance has operationalized training as the amount of training employees receive, the total hours or days of training provided to employees, total dollar amount spent on training, number of workers trained, the percentage of workers who receive training, the type(s) of training provided to employees, and the perceived importance and effectiveness of training (Tharenou etal., Reference Tharenou, Saks and Moore2007). The inconsistency across studies in how training has been operationalized has made it difficult to compare studies and understand what really matters about training when predicting organizational performance. To make matters worse, many studies have used single-item and self-report measures using a single informant to provide data on training (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019).
The way the independent variable is operationalized is a critical factor in organizational-level research because it will influence the results. That is, the nature and strength of the relationship between an I-O intervention and organizational performance will depend on how the I-O intervention is operationalized and measured. It is important to choose measures that are theoretically meaningful and relevant. The measures of training have tended to be very general (e.g., amount of training received) and lacking in specificity and theoretical relevance. As a result, research on training and organizational performance does not tell us much more than simply that “more training is related to greater organizational performance.” This limits our understanding of what it is about training that is most important for organizational performance, why it is important, and how it leads to greater organizational performance. Thus, there needs to be a great deal of thought about how best to operationalize the I-O intervention or practice, and there should be some agreement and consistency across studies about the most valid and reliable operationalization.
Dependent variables
Organizational performance is a multidimensional construct, and it has been measured in a number of ways (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019). As already noted, Tharenou etal. (Reference Tharenou, Saks and Moore2007) found stronger relationships between training and human resource outcomes and organizational performance than for financial outcomes. They also found stronger relationships between training and perceptual measures than objective measures, and most studies (58%) have used subjective measures of organizational performance (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019).
Research on I-O programs and organizational performance should consider using several measures of organizational performance (e.g., human resources [HR] outcomes such as employee attitudes and behaviors, organizational performance outcomes, financial accounting outcomes, and stock market outcomes; Dyer & Reeves, Reference Dyer and Reeves1995). Objective measures are probably more appropriate, however, there is some evidence that perceptual measures of organizational performance are moderately to strongly correlated with objective measures of firm performance (Delaney & Huselid, Reference Delaney and Huselid1996; Wall etal., Reference Wall, Michie, Patterson, Wood, Sheeman, Clegg and West2004). Although publicly reported financial metrics might be the most desirable dependent variables, as suggested by Schneider and Pulakos (Reference Schneider and Pulakos2022), they are the least strongly related to training (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019; Tharenou etal., Reference Tharenou, Saks and Moore2007) and are likely to be the least strongly related to other I-O interventions. The results of research on I-O interventions and organizational performance will depend on how organizational performance is measured, so measurement should be given careful consideration.
Mediating variables
Given how distal organizational performance is from most I-O practices, there are likely to be indirect relationships and mediating variables that intervene between I-O practices and organizational performance. For example, Tharenou etal. (Reference Tharenou, Saks and Moore2007) suggested that the relationship between training and organizational performance is mediated by employee attitudes and human capital, and there is some evidence that HR outcomes mediate the relationship between training and organizational performance. However, most studies on training and organizational performance (51%) have only investigated the direct relationship between training and organizational performance (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019), and few have tested for mediating effects. As a result, we have not learned very much about the mechanisms that explain the relationship between training and organizational performance.
The effects of I-O programs on organizational performance (especially financial performance) are very likely indirect, so it is important to measure mediating mechanisms. Thus, research on I-O interventions and organizational performance should consider mediating variables and test for mediating effects so that we can understand why an I-O practice is associated with organizational performance.
Moderating variables
The relationship between I-O programs and organizational performance might depend on contextual factors. For example, Tharenou etal. (Reference Tharenou, Saks and Moore2007) found that training was more strongly related to organizational performance when it was matched with contextual variables such as an organization’s capital investment intensity and business strategy. Garavan etal. (Reference Garavan, McCarthy, Lai, Murphy, Sheehan and Carbery2021) found that training was more strongly related to organizational performance in low performance-oriented countries than in high performance-oriented countries and in organizations operating in low labor cost countries than in high labor cost economies.
Research on I-O programs and organizational performance should identify and measure contextual variables that might moderate the effect of the program on organizational performance and therefore indicate the boundary conditions for the relationship. The extent to which an I-O program is related to organizational performance will probably depend on contextual variables, and they should be included in I-O research on organizational performance.
Micro–macro integration
One area that research on training and organizational performance has fallen short is integrating individual or micro-level training research with organizational-level or macro-level training research (Tharenou etal., Reference Tharenou, Saks and Moore2007). This requires research that collects data from individuals within and across organizations as well as organizational-level data. Although there have been many studies on training and organizational performance, they have been conducted independently of training research at the individual level of analysis. As a result, these two areas of training research exist independently with little effort to link and integrate them, which has resulted in a micro–macro gap in training research.
Research on I-O practices and organizational performance should develop and expand existing research at the individual level rather than conduct organizational-level research in isolation of research at the individual level. This requires multilevel research that measures variables at the individual level and the organizational level and tests relationships between them in order to demonstrate how they relate to each other such as whether trainee transfer of training is related to transfer of training and performance at the organizational level and across organizations.
Methodological issues
In their review of research on the training–organizational performance relationship, Garavan etal. (Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019) identified many threats to internal, external, construct, and statistical conclusion validity that limit the conclusions that can be made about the relationship between training and organizational performance. Of particular concern is the overwhelming use of cross-sectional (91% of studies) and postpredictive designs (54%) that measure organizational performance prior to training. In addition, most studies (61%) used the same source to measure training and organizational performance at the same time. Not surprisingly, the relationship between training and organizational performance is significantly higher for cross-sectional studies (Garavan etal., Reference Garavan, McCarthy, Sheehan, Lai, Saunders, Clarke, Carbery and Shanahan2019).
As a result of these limitations, it is not possible to make causal conclusions about the relationship between training and organizational performance. Thus, I-O research at the organizational level must avoid these limitations if it is to make meaningful conclusions about the effects of I-O interventions on organizational performance. This will require longitudinal and predictive designs that measure I-O interventions at one point and organizational performance at some future time and, ideally, measure both variables several times.
Conclusion
There is no doubt that I-O research and practice can benefit from more attention to the organizational level of analysis as a frame of reference for I-O research. However, there are important issues to consider when doing I-O research with an organizational mindset if we are to realize the potential benefits and make I-O research more effective in practice and more visible among organizational leaders. This requires research that does much more than simply demonstrating that an I-O practice is positively related to organizational performance.
