Surgical site infection (SSI), the second most common cause of nosocomial infection, accounts for ~16% of all hospital-acquired infections. 1 At the Ottawa Hospital (TOH), patients with an SSI stayed in hospital on average 7 days longer than those without an SSI in 2010. 2 Prolonged hospitalization not only imposes negative consequences to patients but also incurs high opportunity costs by limiting a hospital’s capacity to care for other patients.
Various quality improvement programs have been implemented in attempts to reduce postsurgical complications.Reference de Vries, Ramrattan, Smorenburg, Gouma and Boermeester 3 – Reference Guillamondegui, Gunter and Hines 9 Among these programs, the American College of Surgeons National Surgical Quality Improvement Program (NSQIP) is considered the preeminent surgical quality improvement program.Reference Dellinger, Hausmann and Bratzler 5 The NSQIP provides detailed data necessary to measure and monitor SSI rates monthly; these data can be used to estimate the direct and indirect costs incurred by surgical patients. The NSQIP was implemented at TOH in May 2010 as the first phase of the hospital’s surgical quality improvement program (SQIP). Outcome data have been collected on ~20% of all surgical procedures performed at TOH through the NSQIP Essentials Program. The Comprehensive Unit-based Safety Program (CUSP) was initiated in March 2013 as the second phase of the TOH SQIP. The NSQIP is not an intervention, but its data provide insight for CUSP in designing quality improvement initiatives, and it enables the outcomes to be monitored and evaluated effectively. The Ottawa Hospital has formed 17 multidisciplinary CUSP teams and 7 CUSP working groups by surgical subspecialty, intervention, and campus to initiate surgical best practices to reduce surgical complications, including SSIs. These teams and working groups have implemented 29 major perioperative quality improvement interventions. In this study, we estimated the hospital costs and savings associated with TOH’s quality improvement program, namely NSQIP and CUSP, and we performed a return on investment (ROI) analysis based on total hospital investment in SSI prevention compared to savings from averted SSI cases.
Methods
Study setting and population
This retrospective study used aggregated records for all TOH patients who underwent surgical procedures between April 2010 and January 2015. No individual patient data were used. The Ottawa Hospital is a tertiary-care teaching institution containing 1,118 beds.
Data source and costing
Data used for this study are obtained from TOH Data Warehouse, a relational database containing the operational information of each of TOH’s campuses. In this study, hospital costs for each inpatient encounter were identified within the case-costing system of TOH Data Warehouse, a standardized case-costing methodology developed by the Ontario Case Costing Initiative 14 based on the Canadian Institute for Health Information Management Information Systems guidelines. 15 The case-costing system links financial, clinical, and patient activity information stored within the Data Warehouse to define intermediate products, such as nursing time, medications, and laboratory tests. The total hospital costs were equal to the sum of the direct and indirect costs for each intermediate product used during an encounter for each patient.
The Data Warehouse houses administrative data dating back to 1996; however, SSI cases have only been reliably collected through NSQIP since March 2010. We used monthly incidence rate of SSI, incurred hospitalization costs, and average hospital length of stay from April 2010 to January 2015 from the NSQIP system. Surgical patient information and hospital costs were aggregated per month by surgical specialty (ie, department or division) in 3 broadly planned admissions categories: inpatients, overnight patients (only 1 night hospitalization) and day surgery patients, and SSI infection status. Attributable cost of SSI was calculated independently for each of the 3 surgical categories and then combined using a weighted average based on the proportion of surgical patients in each category.
The cost of NSQIP is based on an annual fee to the hospital, which includes licensing, information technology and maintenance, and designated staff to input health records information into the data repository system. Although NSQIP monitors between 250 and 500 complications,Reference Thanh, Baron and Litvinchuk 7 guidance from the hospital quality improvement teams led us to conclude that the decision to invest in NSQIP is based on the target outcomes of 19 major postoperative complications. SSI is one of these outcomes. Our model attributes 1 of 19 of the total annual cost of NSQIP (5.3%) to SSI prevention, as shown in the annual cost column reported in Table 2 hospital costs.
The cost of CUSP is based on the annual average cost to the hospital of managing the surgical complications prevention initiatives, which includes costs of materials introduced to the surgical protocols and postsurgical care, staff training and monitoring time, and salaries of the designated quality improvement coordinators.
Analysis
In the primary analysis for the return on investment model, we calculated the annual savings from averted SSI cases (Table 1). Savings were calculated as a product of the cost per SSI case, and the change in surgical specialty incidence of SSI from the previous year. Surgical department level costs are the share of total annual costs to the hospital to manage NSQIP and CUSP as a proportion of number of surgical patients within that specialty per year. Annual net savings are the difference in total savings from averted SSIs minus the total investment. All costs are adjusted to 2016 Canadian dollars. An annual discount rate of 1.5% was applied to both investment and savings. The return on investment analysis calculates the cumulative net saving over several years and divides the savings over the cumulative spending on NSQIP and CUSP initiatives to provide a cost–benefit ratio as dollar return per dollar spent.
Table 1 The Ottawa Hospital Surgical Patient Average Hospital Costs and Length of Stay, by SSI Outcome
Note. SSI, surgical site infection; CI, confidence interval.
Table 2 Return on Investment Model Input Parameters
Note. CI, confidence interval.
We also introduced several scenario analyses to the primary analysis to determine whether the results indicate the same direction and scale of return on investment if we expanded the analytical perspective.
In the first scenario, we asserted the cost of new admissions for every averted SSI case. The Ottawa Hospital typically operates at full capacity, and consequentially, an averted SSI case represents a now open inpatient space that can be filled. In our first scenario, therefore, we applied the average daily cost of an admitted patient to TOH for the number of added days associated with an SSI case.
In the second scenario, we extended the implications of new admissions by adding a fixed payment to the hospital from the health system as added revenue from increasing total patient capacity of the hospital. The manner by which the health system calculates hospital operational reimbursements is complex and is not driven by a single standardized unit of improved patient care or total patients seen. Therefore, we assumed a wide range of revenue received by the hospital per patient based on expert consultation.
In the third scenario, we considered potential savings from averted SSI cases, which expanded the analytical perspective to include both hospital and axillary care institutions (eg, rehabilitation, long-term care, etc.). An SSI can require additional care following discharge, particularly in the form of home care services that are not captured in the administrative data base search.Reference Berwick, Calkins, McCannon and Hackbarth 4 We considered the added costs of postdischarge patient care assuming that this care falls within the cost perspective of the hospital.
We performed a one-way sensitivity analysis on all parameters within the base case model as well as the scenario analyses. The results of the sensitivity analysis are presented in a tornado plot to show what parameters contribute most significantly to model uncertainty.
Historical data on SSI incidence were based on limited chart linkages to administrative costing data, leading to the introduction of NSQIP at TOH. Therefore, we do not have reliable preintervention incidence data from which we could calculate the attributable reduction in SSI incidence based on a preintervention incidence trend. We performed a secondary sensitivity analysis that specifically addressed this uncertainty in preintervention incidence reduction in order to calculate the minimum level of improvement we would need to see for the quality improvement program to be economically attractive.
Important to a decision analysis, we generated a ‘ROI frontier’ analysis wherein we calculated the return on investment for every potential value of average annual SSI incidence reduction based on a cumulative ROI ratio after 6 years (2010–2016) and the associated annual SSI incidence improvement.
Finally, we performed a probabilistic sensitivity analysis (PSA) wherein all the parameters are varied according to its confidence interval and distribution, using a Monte Carlo approach with 30,000 iterations.
Results
Our base-case analysis shows that TOH invested a cumulative US$624,384 in surgical quality improvement programs targeting SSI incidence from 2010 to 2015. In that same time period, SSI incidence has decreased by 2.88% (average of 0.48% annually) resulting in a cumulative savings from averted SSI cases of US$1,423,460. This represents a return of US$3.07 per dollar invested.
The observed positive return on investment is not uniform across surgical specialties. Figure 1 shows the cumulative ROI per year by surgical specialty in stacked bars, with the single curve representing hospital-wide ROI. Return on investment calculations up to year 2016–2017 are based on a linear average annual SSI incidence rate per specialty, while years following 2016 are extrapolating the predicted returns assuming the same trend into the future. Table 3 displays the surgical specialty-level SSI incidence and monthly capacity (ie, number of surgeries).
Fig. 1 Cumulative return on investment by department.
Table 3 Department- or Division-Level Incidence and Surgical Capacity
Note. SSI, surgical site infection.
a Defined as the average full-time operation complement of patients seen by the surgical department.
The PSA shows a mean ROI ratio of 2.28 (95% confidence interval [CI], −0.67 to 7.37). In Figure 2, the y-axis represents probabilities that the quality improvement program offers a positive return on investment, whereas the x-axis shows the annual percentage reduction in SSI incidence. Using the assumption that SSI incidence has reduced by 0.48% per year over 6 years, the probability that the quality improvement programs to date have netted a positive return on investment is 43%.
Fig. 2 Probability of positive return on investment, probabilistic sensitivity analysis.
Including the cost of new admissions (scenario 1) into the model significantly reduced the expected ROI of the quality improvement program to 0.49 (95% CI, −1.50 to 1.40) due to the lower expected savings per patient. The other 2 scenarios, including (scenario 2) hospital revenue for new patients and (scenario 3) expanding costing perspective to include cost of posthospitalization care for SSI patients, had a significant positive impact on expected ROI, as both assigned higher cost per SSI case and therefore a higher net savings per case averted. Under either scenario, there is a high level of confidence that the return on investments to date net a greater than 2-to-1 return on investment of 3.06 (95% CI, 2.49–3.30) for scenario 2 and 4.12 (95% CI, 2.27–5.02) for scenario 3]. The combination of all 3 scenarios results in a similar probability of being a positive ROI as the primary analysis.
The one-way sensitivity analysis (Fig. 3) revealed that the cost per SSI case introduces the highest level of uncertainty into the model. The second-most influential variable on return on investment results was annual SSI incidence reduction followed by the annual cost of CUSP and NSQIP, respectively.
Fig. 3 One-way sensitivity analysis, return on investment.
Discussion
Our study shows encouraging evidence that investments in NSQIP and CUSP have netted cost savings that may already be producing a positive return on investment at TOH. This return on investment is expected to continue in the near term assuming the reduction in SSI rate continues.
The return on investment is not uniform by surgical specialty. This variability is mostly driven by the capacity of the department or division and the relative potential of SSI improvement. Specialties with much lower numbers of surgeries per year may experience a decrease in SSI incidence rate, but because the savings are in absolute dollars per patient, they may not offer sufficient savings for the share of investment going to that particular specialty. Additionally, specialties with historically high SSI incidence, can experience significant reductions in a relatively short time (eg, vascular surgery) leading to a significant ROI in those specialties. For other specialties with historically low SSI incidence, the rate of improvement is unlikely to be great due to the type of surgery performed or the level of care already provided.
The primary return on investment analysis assumes the average annual reduction in SSI incidence observed at TOH (0.48%) is fully attributable to the quality improvement programs initiated. Without reliable incidence data prior to NSQIP, it is difficult to test the true treatment effect of NSQIP and CUSP. However, we can estimate the minimum incidence reduction necessary given the fixed investments to date. Figure 4 displays the results of an ROI frontier analysis that finds that the minimum annual SSI incidence reduction necessary for TOH’s current investments to net a positive ROI (>1) is 0.25%. Our ROI estimate used the observed annual reduction in SSI incidence of 0.48%, leaving a 0.23% gap between the minimum reduction necessary to achieve positive ROI. Assuming the preintervention average annual reduction of SSI is <0.23%, the investments to date have netted at least a positive return on investment as of 2016.
Fig. 4 Return on investment by average annual surgical site infection (SSI) reduction using the frontier curve.
An important consideration for decision makers when considering long-term investment strategies is that improvements in SSI incidence is unlikely to remain relatively linear, and instead is likely to plateau. As shown in Table 3, those with historically low SSI incidence are already showing difficulty in continuing to improve outcomes. This is likely due to the remaining SSI cases representing a relatively small absolute number of patients and may represent the most difficult cases for which a care provider can avoid infection.
The PSA is used to determine how much variability we find in the result based on the level of uncertainty in the model parameters. The analysis finds a relatively low probability (43%) that the quality improvement program to date has a positive return on investment for the hospital. This is predominantly due to the wide confidence interval of cost per SSI case, partly due to the significant variation in the severity of SSI that requires different levels of hospital resources. Previous research has shown the significant impact of SSI on a patient’s length of stay and overall costs of care,Reference de Vries, Ramrattan, Smorenburg, Gouma and Boermeester 3 , Reference Berwick, Calkins, McCannon and Hackbarth 4 so there is a reasonable expectation that the confidence interval overstates the possibility that patients with SSI are in fact cost saving. For this reason, we believe the ROI frontier curve is more directly applicable to the hospital’s consideration of whether the investments to date have already generated a positive return on investment.
Few published studies have assessed the cost or cost-effectiveness of NSQIP and/or CUSP-like interventions to date. The assessment methods used in previous studies were extremely varied. Only 2 studies conducted cost-effectiveness analysesReference Thanh, Baron and Litvinchuk 7 , Reference Hollenbeak, Boltz and Wang 8 ; 6 studies performed a cost analyses.Reference Thanh, Baron and Litvinchuk 7 , Reference Guillamondegui, Gunter and Hines 9 – Reference Tepas, Kerwin, deVilla and Nussbaum 13 Among these studies, 6 studiesReference Thanh, Baron and Litvinchuk 7 – Reference Guillamondegui, Gunter and Hines 9 , Reference Ceppa, Pitt and House 11 , Reference Tepas, Kerwin, deVilla and Nussbaum 13 found that quality improvement programs resulted in savings to the hospital; however, only one study detailed the type of interventions that took place other than NSQIP.Reference Thanh, Baron and Litvinchuk 7 In 4 cases, the program under evaluation was NSQIP itself.Reference Hollenbeak, Boltz and Wang 8 – Reference Osborne, Nicholas, Ryan, Thumma and Dimick 10 , Reference Tepas, Kerwin, deVilla and Nussbaum 13 Also, 2 studies used the American College of Surgeons (ACS) NSQIP ROI calculator to determine net savings.Reference Hollenbeak, Boltz and Wang 8 , Reference Tepas, Kerwin, deVilla and Nussbaum 13 This online calculator uses an average cost of SSI based on the entire ACS hospital network, making it highly unreliable to a specific hospital’s circumstance. Only 4 studiesReference Thanh, Baron and Litvinchuk 7 – Reference Osborne, Nicholas, Ryan, Thumma and Dimick 10 performed a sensitivity analysis, of which only 1 studyReference Osborne, Nicholas, Ryan, Thumma and Dimick 10 reported the possibility that savings from NSQIP could be potentially not cost saving.
Methodologically, the closest publication to our own analysis is an Albertan study across 5 hospital sites that estimated a NSQIP and associated SQIP initiatives’ return on investment of $4.3 per dollar spent. The study focused their estimates of SSI costs and intervention effects on specific departments that were targeted with quality improvement initiatives.Reference Thanh, Baron and Litvinchuk 7 The difference in their estimates can be attributed to the higher attributed cost of SSI and added benefits from reductions in UTI and blood transfusions, which increases the potential benefit of the intervention, as well as focusing on priority departments. As our study has shown, departmental returns on investment can vary dramatically. Our findings are well complimented in showing that NSQIP, in combination with well-designed quality improvement initiatives, is likely to produce positive returns on investment.
Our study is the first to perform a cost analysis with a detailed and generalizable methodology for calculating the cost of an SSI that incorporates the costs of both NSQIP and the associated quality improvement program interventions, and presents results at the departmental level. We also introduce the potential of a ROI frontier curve, which we believe is a valuable decision-making tool for other hospitals to consider.
Our study has 2 notable limitations in the study and model design. The first is that by using aggregate-level information from the hospital, we have high uncertainty in the outcomes, primarily driven by the estimated per-patient cost of SSI. Although we present a full analysis that includes the statistical uncertainty from the available data, the extreme range of potential costs of SSI may not be a reasonable reflection of the cost of SSI that we would see if we controlled for other confounding factors such as patient characteristics, SSI severity, and type of surgery (eg, differentiating by urgent or elective surgery).
The second limitation is that we cannot reasonably estimate the attributable effect of quality improvement programs such as CUSP on SSI incidence. It is possible that exogenous effects such as improved staff experience, change in patient safety culture, improvements in surgery techniques that are less invasive, changes in hospital infrastructure, and unknown ad hoc initiatives have some impact on overall SSI incidence. To address this limitation, we conducted a scenario analysis that considers all possible attributable effects of SSI incidence on the hospital’s ROI. The ROI frontier curve can be a valuable decision-making tool for estimating the target SSI incidence change necessary for the current annual investment to offer a positive return to the institution.
This study has some limitations. The ROI estimates observed in this study may not be easily generalizable to other hospitals due to standard eccentricities of a given hospital’s operations, internal costs, and the suite of interventions they select to implement. The methodological approach, however, was designed to apply a return on investment framework to any Canadian hospital setting in a manner that is replicable and generalizable to the question and available data.
In conclusion, this study shows how an institutional return on investment framework can be applied to quality improvement initiatives at a hospital. The investments in NSQIP and CUSP to date at TOH likely have a substantial positive return on investment of US$3.07 for every dollar invested, though the ROI estimates rest on several assumptions, primarily the cost of an SSI case and share of incidence reduction attributable to NSQIP. Although this study does present useful methods for evaluating return on investment in the face of this limitation, decision makers should encourage a rigorous program evaluation of the costs and attributable effectiveness of the quality improvement initiatives to ensure that such initiatives are improving the quality of care and continue to provide positive return on investment.
Acknowledgments
We thank Deanna Rothwell, the Manager of Performance Measurement at The Ottawa Hospital for providing data required for this study. We also wish to thank all reviewers for their insightful comments on the manuscript.
Financial support
This study was funded by Health Quality Ontario.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
