Outcome measures in health care play a pivotal role in quantifying the ability of an organization to provide high-quality healthcare. Healthcare-associated infection (HAI) measures, in particular National Healthcare Safety Network (NHSN)–defined central-line–associated bloodstream infection (CLABSI) rates, are becoming increasingly important as the Centers for Medicare and Medicaid Services (CMS) and private insurers use these measures in pay-for-performance programs such as the Hospital-Acquired Conditions Reduction program and the 2015 Value-Based Performance program.Reference Rajaram, Barnard and Bilimoria 1 The majority of US states mandate public reporting of CLABSI data and publish these data in hospital report cards available to consumers, healthcare providers, and hospital administrators for comparison of hospital performance in quality of care. 2 However, for an outcome measure to adequately serve this purpose, it needs to reflect the truth, be feasible, and have the power to discriminate between facilities.Reference Sexton, Chen, Moehring, Thacker and Anderson 3 Several studies have shown that the NHSN CLABSI rates (1) do not necessarily reflect the truth, (2) are subjective and resource-intensive, and (3) are therefore a questionable choice for such a highly weighted outcome measure.Reference Sexton, Chen, Moehring, Thacker and Anderson 3 – Reference Mayer, Greene, Howell, Ying, Rubin, Trick and Samore 7 Another potential major limitation of NHSN CLABSI as a quality measure is that uniformly low CLABSI rates nationally—including frequent “zeros”—may no longer allow meaningful comparisons between hospitals, ie, this outcome measure may lack the power to truly discriminate between hospitals.
In this study, we investigated a new HAI outcome measure, hospital-onset bacteremia (HOB), defined as a positive blood culture obtained ≥48 hours after hospital admission. Compared with CLABSI, HOB is objective, simple to understand, easily automated, and easier to collect and, thus, is time saving. In addition, HOB is a more global or inclusive measure of HAI-related quality because it incorporates bacteremia as a result of any healthcare-associated infection (eg, urinary tract infection or pneumonia) and not just CLABSI.
The first study hypothesis is that changes in HOB rates are associated with changes in CLABSI rates, meaning that changes in HOB would reflect changes in CLABSI. Thus, HOB should be used as a CLABSI surrogate because it is a more inclusive measure than CLABSI. The second study hypothesis is that HOB is a more frequent event than CLABSI and thus has greater power to discriminate between (ie, “rank”) hospitals.
METHODS
In this multicenter ecological study, hospitals were recruited through the SHEA Research Network. The SHEA Research Network is a consortium of >200 hospitals conducting multicenter research projects in healthcare epidemiology.Reference Drees, Pineles, Harris and Morgan 8 , Reference Morgan, Meddings and Saint 9 Facilities within the United States and Canada with adult, pediatric, or neonatal ICUs were invited to participate. Each center obtained approval from its respective institutional review board.
Study variables were defined as follows. CLABSI was defined as a primary bloodstream infection in a patient with ≥1 central line within the 48-hour period prior to the onset of the bloodstream infection, and the bloodstream infection was not related to any infection at other foci, per CDC definitions. 10 HOB was defined as a positive blood culture for any organism from any cause (including contaminants and repeat positive blood cultures) sent from the ICU and taken ≥48 hours after admission to hospital. HOB rate was defined as the number of HOBs divided by the number of ICU patient days. The total number of blood cultures obtained included all blood cultures, positive and negative, sent from the ICU for each study month.
Data Collection
Each participating hospital contributed monthly aggregate data for each ICU for the number of CLABSIs, central-line days, HOBs, ICU patient days, and total number of blood cultures obtained from January 2012 to December 2013. CLABSI determination was performed by each hospital’s infection prevention program, independent of this study, by conducting chart review using standard CDC NHSN definitions and reporting methods. The components of the HOB outcome measure were retrieved in an automated fashion directly from hospital microbiology and admission-transfer-discharge databases without medical record review. The ICU-type was also collected using CDC-NHSN classification. 10 Each participating hospital completed an on-line survey to assess hospital and ICU level factors (see Online Supplementary Appendix). Questions included the number of infection preventionists at the hospital and the estimated time spent by infection preventionists on CLABSI surveillance.
Statistical Methods
Association between HOB and CLABSI
We tested the association between HOB and CLABSI using a mixed-effects Poisson regression model. Candidate predictors included HOB rate, time period (month and year), hospital, ICU type, and total number of blood cultures obtained. Backward selection for best fit model, using the deviance information criterion, combined with clinical judgment, with CLABSI rates as an outcome was performed. The total number of blood cultures obtained was expressed as a rate per 1,000 ICU patient days and was included because it was considered an important potential confounder. The ICU was included as a random effect, to account for correlation of observations within the ICU. HOB rate and total number of blood cultures per ICU patient days were included as fixed effects. The overdispersed distribution of CLABSIs was adjusted using additive overdispersion.Reference Hadfield 11 These analyses were performed in the R programming language using the MCMCglmm package. 12
Discrimination between ICUs
We assessed the ability of HOB and CLABSI to discriminate between different ICUs of the same type using 2 methods: (1) standardized infection ratios (SIRs) and (2) proportion of ICU months with zero CLABI and zero HOB.
For method 1 we used indirect standardization methods similar to those used by CMS on the Hospital Compare website, and we benchmarked each ICU against similar types of ICU within the cohort. 2 For each ICU type (eg, medical ICU [MICU], surgical ICU [SICU], etc), we summed the total number of patient days and the total number of positive blood cultures for all ICUs of that type, and divided the total number of positive blood cultures by the total number of patient days to get the “benchmark” HOB rate for that type of ICU. For each ICU in the study, the number of expected HOB was calculated using the benchmark rate and observed patient days. This observed number of HOB was then divided by the expected number to calculate an HOB SIR. This procedure allowed for the comparison of ICUs with different numbers of patient days; a MICU with a higher number of patient days would be expected to have a higher number of HOBs than another MICU with fewer patient days. The same procedure was used to calculate CLABSI SIRs. Poisson 95% confidence intervals (CI) around each SIR were calculated and interpreted as follows: An SIR 95% CI that includes 1 means that the ICU rate is the same as expected for that type of ICU; >1 indicates that ICU has a higher than expected rate; and <1 indicates a lower than expected rate. The proportion of ICUs whose SIR and 95% CIs included 1 were calculated and compared for CLABSI and HOB using χ2 (or Fisher’s exact) test. We also calculated SIRs for each of the hospitals for overall CLABSI and HOB rates using similar methods.
Using the second method, we assessed for a “ceiling effect” by calculating the percentage of the total ICU months with the minimum possible number of CLABSIs and HOBs (eg, zero). The term ceiling effect is used when the performance of a large proportion of subjects for a given measure is as “good” as possible.Reference Taylor, Redden and Dalbeth 13 The presence of the ceiling effect implies that power to discriminate is compromised and further improvement in performance cannot be captured. We compared the ceiling effect between CLABSI and HOB by comparing the proportion of ICU months with zero CLABSI to the proportion that had zero HOB, using a χ2 test. These analyses were performed using SAS 9.3 (SAS Institute, Cary NC).
RESULTS
Intensive care units from 16 hospitals in the United States and Canada participated in the study. Of the 16 hospitals, 13 were academic hospitals. The number of beds was >500 in 10 hospitals, 300–500 in 4 hospitals, and 100–300 in 2 hospitals. The average numbers of beds in adult and neonatal ICUs were 17.4, and 37.3, respectively. The average numbers of infection preventionists were 5.1 per hospital and 1.1 per ICU. Infection preventionists spent an average of 16.6 hours per week on CLABSI surveillance (an average of 2.9 hours per ICU per week). Over the 2-year study period, there were 982,609 ICU patient days, 475,420 central-line days, and 157,383 total blood cultures obtained, with 11,280 HOBs and 663 CLABSIs reported; CLABSIs represented approximately 6% of the overall HOB. Table 1 shows ICU type and number as well as CLABSI and HOB rates and ranges for the participating ICUs.
TABLE 1 ICU Types, Frequencies, and Rates of Central-Line–Associated Bloodstream Infection (CLABSI) and Hospital–Onset Bacteremia (HOB)
a CLABSI rate is expressed per 1,000 central-line days.
b HOB rate is expressed per 1,000 ICU patient days.
Correlation between CLABSI and HOB Rates
The best, most parsimonious model had the HOB rate and the rate of total number of blood cultures obtained as independent variables. Adjusting for the rate of blood cultures obtained, HOB was associated with CLABSI; an increase in the absolute rate of 1 HOB per 1,000 ICU patient days was associated with a relative increase of 2.5% in the CLABSI rate (P<.001). The regression equation is as follows: log[CLABSI rate/(1−CLABSI rate)]=−1.639+(0.024×HOB per 1,000 patient days)+(−0.47991× total blood culture per 1,000 patient days). For example, an ICU with an increase in their HOB rate from 20 HOB per 1,000 ICU patient days to 30 HOB per 1,000 ICU patient days would expect to see an associated 27.7% increase in their CLABSI rate, eg, from 2 CLABSI per 1,000 central-line days to 2.54 CLABSI per 1,000 central-line days.
Discrimination between ICUs
Of 80 participating ICUs, 20 (25%) had CLABSI rates that could be distinguished from the same type of ICU (SIR with 95% CI that does not include 1), while 60 of 80 ICUs (75%) had HOB rates that were different from the rate for the same type of ICU (P<.001). Figure 1 shows the SIRs for the 2 most common ICU types: MICU and neonatal ICU [NICU].
FIGURE 1 Standardized infection ratios (SIRs) for central-line–associated bloodstream infection (CLABSI) and hospital–onset bacteremia (HOB) for each of the medical intensive care units (MICUs) and neonatal ICUs (NICUs). The vertical line at 1 represents the reference or null value: where the expected rate (study benchmark) of CLABSI or HOB for each MICU or NICU lies (SIR=1). The filled-in square represents the HOB rate and the filled-in circle represents the CLABSI rate. The horizontal line though each symbol represents the 95% confidence interval around the perimeter. Those that lie to the right of the SIR 1 reference line have a higher-than-expected number of CLABSIs or HOB (red; worse than the study benchmark). Conversely, those that lie to the left have a lower-than-expected number of CLABSIs or HOB (green; better than the study benchmark). Those that include the expected number of CLABSI or HOB include the SIR reference line and are shown in orange.
Pooling the CLABSI and HOB data from all ICUs per hospital, 9 of 16 hospitals (56.3%) had CLABSI SIRs that included 1, and 2 of 16 hospitals (12.5%) had HOB SIRs that included 1 (P =.02). CLABSI rates were zero (ie, achieved the ceiling effect) for 71.7% of individual ICU months (1,376 of 1,920) compared to HOB rates, which were zero in 221 of 1,920 ICU months (11.5%; P<.0001).
DISCUSSION
We collected CLABSI rates and calculated HOB rates for 80 ICUs in 16 hospitals within the United States and Canada. We found that changes in HOB rates and CLABSI rates were significantly associated, demonstrating that HOB has merit and should be explored as a surrogate marker for CLABSI. We have also shown that HOB is much better at discriminating between ICU performances than CLABSI.
HOB is a more global measure of preventable HAIs than CLABSI; it is inclusive of not only CLABSI but also bacteremias from other causes such as urinary tract infection and pneumonia. Potential advantages of HOB over CLABSI include the objectivity of the definition; it does not require chart review of the bacteremia and can be easily obtained in an automated manner from hospital databases. We found HOB rates to be significantly associated with CLABSI rates over time; for every change of 1 HOB per 1,000 ICU patient days, there was a 2.5% change in CLABSI rate. This association has also been demonstrated in a previous single-center study, in which a 5.1% decrease in CLABSI post intervention to prevent CLABSI was associated with a 2.7% decrease in HOB.Reference Leekha, Li and Thom 14 This finding provides support for the premise that HOB could be used in place of CLABSI as a more inclusive HAI outcome measure but still reflect changes in CLABSI rates.
The second major finding of this study was that HOB is much better at discriminating between ICU performances than CLABSI. To demonstrate this point, SIRs were calculated and interpreted in a similar way to the method CMS uses on the Hospital Compare website. 2 ICUs were assigned red, yellow, or green “traffic light” colors, meaning they were worse than, the same as, or better than other ICUs of the same type, respectively. The majority of CLABSI SIR confidence intervals (60 of 80; 75%) included 1, meaning that the majority of CLABSI SIRs could not be discriminated from an SIR of 1 or from the average performance of same type ICUs. For HOB SIRs, only 20 of 80 (25%) included 1, meaning that the majority could be discriminated from the average performance of same type ICUs. The power to discriminate between SIRs increased with increases in the expected outcome frequency (“sample size”). Thus, the greater the expected numbers of outcomes (CLABSIs or HOBs), the narrower and more precise the confidence interval and the greater the likelihood of discriminating between 2 SIRs (Figure 1).Reference Greenland 15 , Reference Gordon 16 With so many ICUs receiving the same “traffic light” color for CLABSI and the large overlapping confidence intervals showing little discrimination, it is difficult to truly discern poor and good quality; thus, CLABSI may not be an ideal measure for public reporting. Notably, some ICUs had different colored SIRs for CLABSI and for HOB, eg, MICU E. Possible reasons for this difference may be the inherent lack of objectivity of the CLABSI measure and the potential for over- or underreporting. Also, because the expected number of CLABSIs is so few, even 1 or 2 additional CLABSIs could result in an SIR changing from green to red. The NHSN CLABSI SIR has a strong weight in domain 2 of the CMS Hospital-Acquired Conditions Reduction program and is also included in the Value-Based Purchasing program. 17 Given the lack of power of CLABSI SIR to truly discriminate between ICUs in our study, CLABSI appears to be a poor choice for an HAI outcome measure used in external benchmarking, and hospitals could be unfairly financially penalized as a result.
One reason that the CLABSI outcome measure lacks power to discriminate is the infrequency of the occurrence of CLABSI. In fact, 71.7% of individual ICU months had zero CLABSIs; conversely, only 11.5% ICU months had zero HOB. This study shows that CLABSI is subject to ceiling effects, meaning that it is often at the lowest or “best performance” level. There are likely numerous reasons for the infrequent number of CLABSIs seen in our study and in the real world.Reference Vallés, León and Alvarez-Lerma 18 First, there is likely a true decrease in CLABSI resulting from improvements in infection prevention in the last decade. In addition, it is possible that, due to the significant financial and reputational repercussions associated with higher than expected CLABSI rates and lack of objectivity in the application of the definition, there may be some intentional or unintentional underreporting of CLABSI rates.Reference Sexton, Chen, Moehring, Thacker and Anderson 3 , Reference Lin, Hota and Khan 4 , Reference Niedner 6 , Reference Vallés, León and Alvarez-Lerma 18 , Reference Haut and Pronovost 19 Also, smaller hospitals with fewer ICU beds and fewer central-line days may also have frequent zeros. In our study, CLABSI represented only ~6% of HOB. A hospital may have preventable bacteremia, but by only measuring those cases that are associated with central lines (ie, CLABSI), further improvement cannot be measured due to ceiling effect, and may lead to a false sense that no opportunity for improvement exists.Reference Yokoe, Anderson and Berenholtz 20
An additional important benefit of HOB over CLABSI is that it is less resource-intensive than CLABSI. In our study, infection preventionists spent an average of 16.6 hours per week on CLABSI surveillance, which is predominantly manual chart review. For HOB surveillance, there is no manual chart review required because all bloodstream infections are included, regardless of etiology. This move toward laboratory-based surveillance is supported by the recent introduction of laboratory-based surveillance for Clostridium difficile infection and methicillin-resistant Staphylococcus aureus bacteremia by the NSHN.
The limitations of this study include the retrospective nature of the data. However, bloodstream infections already classified as CLABSIs were included in an effort to represent real-world conditions as much as possible, and these same CLABSIs would be reported to NHSN. Further, the retrospective nature of this study also ensured that CLABSI determination was not affected by the study itself. We used internal sample-derived benchmarks to calculate the SIRs for both HOB and CLABSI. We included multiple positive blood cultures from the same infection episode more than once and included “contaminants” in our HOB rate to make this measure as objective and simple as possible. Laboratory-based algorithms could be used in the future to delete repeated positive blood culture data. Although “contaminants” such as single episodes of coagulase-negative staphylococcus bacteremia may not represent HAI, it is likely that good blood-drawing technique with adequate attention to sterile technique would result in low rates of these contaminants so the presence of these organisms may in a sense represent poor quality of care. We did not collect data on the microbiology or etiology of hospital-onset bacteremia and were therefore unable to identify specific areas for improvement. However, the aim of this particular study was to evaluate this metric for external reporting rather than understand the causes of bacteremia in the hospital. Finally, both CLABSI and HOB are subject to the possibility of surveillance bias, ie, the harder you look the more you find.
A potential challenge for implementation of the HOB measure is the lack of baseline data to derive national SIR benchmarks for HOB. However, if adopted, similar to other new outcome measures, hospitals could submit HOB data to the NHSN during a baseline period of data collection only (ie, without public reporting). Based on these data, benchmark HOB rates could be established for future SIR calculations. In summary, in this multicenter study, we showed that HOB rates are strongly associated with CLABSI rates and that HOB has much better power than CLABSI to discriminate between ICU HAI-related quality performances. These results, in addition to the objectiveness, simplicity, and global nature of the HOB measure may make it more attractive than CLABSI as an outcome measure of hospital quality of care. However, this study also highlights areas worthy of future research prior to utilization of HOB as a quality outcome measure. These include identifying the causes of hospital-onset bacteremia, understanding how often and for what indication duplicate positive blood cultures occur, appropriate ways of risk adjustment of HOB rates, and assessment of hospital-onset bacteremia in non-ICU locations and in the community hospital setting, which was underrepresented in our study.
ACKNOWLEDGMENTS
Financial support. This study was supported by the Society of Healthcare Epidemiology of America. KAT is supported by the National Institutes of Health (NIH) Career Development (grant no. 1K23AI08250-01A1). ADH is supported by the NIH Epidemiology of Emerging Pathogens among Hospitalized Patients (grant no. 5K24AI079040-05) and by the Agency of Healthcare Research and Quality (AHRQ) (grant no. 1 R01 HS022291-01).
Potential conflicts of interest. All authors report no conflict of interest.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit http://dx.doi.org/10.1017/ice.2015.261
