Central-line–associated bloodstream infections (CLABSIs) are responsible for substantial morbidity and mortality among hospitalized patients. Patients with CLABSIs are at a higher risk of death, have longer hospital stays, and incur more healthcare costs than patients without CLABSIs.Reference Stevens, Geiger, Concannon, Nelson, Brown and Dumyati 1 Since January 2012, hospital reimbursement by the Centers for Medicare and Medicaid Services (CMS) has depended on public reporting of CLABSI rates. CMS hospitals use the operational system of the Centers for Disease Control and Prevention (CDC) National Healthcare Safety Network (NHSN) to facilitate reporting. 2
The CDC uses risk adjustment to more fairly compare CLABSI rates across hospitals. Until 2017, the CDC NSHN adjusted CLABSI rates only by type of intensive care unit (ICU). In 2017, the CDC added hospital size (ie, number of licensed beds) and medical school affiliation as additional risk-adjustment variables. 3 However, neither of these CDC models adjust for individual patient level factors, including comorbid conditions. We hypothesized that risk adjustment could be improved by including demographics and comorbid conditions from electronically available hospital discharge codes.
METHODS
Using a cohort design, we retrospectively analyzed ICU patients admitted between January 1, 2012, and December 31, 2013, to 22 US hospitals. Facilities were recruited as part of a partnership between Premier, Inc, the Society for Healthcare Epidemiology of America (SHEA) Research Network, and the University of Maryland School of Medicine. Institutional review board and facility consent were obtained from facilities that voluntarily participated in the study.
Using Premier’s Quality Advisor database, we obtained demographic and International Classification of Diseases, Ninth Edition, Clinical Modification (ICD-9-CM) discharge codes for each adult ICU patient. Patients with CLABSIs were identified by trained infection preventionists at each hospital using CDC NHSN definitions.Reference Horan, Andrus and Dudeck 4 We also obtained information on the size of the hospital (ie, number of beds) and whether the hospital was associated with an academic medical school.
Risk-adjustment models were built using discrete survival analysis, a method that accounts for time at risk.Reference Allison 5 Specifically, acquisition of CLABSI on each day in the ICU was used as the outcome of a binary regression model with a complementary log-log link. A random intercept for hospital was included in the model to account for the clustering of patients within hospitals.
We constructed 2 models: (1) a model containing only ICU-type (ie, CDC methodology prior to 2017) and (2) a model containing ICU-type plus patient case-mix variables. For the latter model, we identified candidate comorbidity variables using expert consensus, which has been reported elsewhere.Reference Harris, Pineles and Anderson 6 Using a modified Delphi method, 9 infectious disease and infection control experts were asked to rate the 35 comorbid conditions found in the Charlson and Elixhauser comorbidity indices from 1 (not at all related) to 5 (strongly related), based on perceived relatedness to CLABSI. These experts rated the following 14 conditions in terms of causality with CLABSI as 3 (somewhat related) or higher: coagulopathy, dementia, diabetes without complications, diabetes with complications, drug abuse, hemiplegia or paraplegia, HIV/AIDS, lymphoma, malignancy, solid tumor with metastasis, severe liver disease, obesity, renal disease, and weight loss (malnutrition). These 14 conditions (identified using ICD-9-CM codes), along with ICU type, age, gender, race, hospital size, and medical school affiliation were entered into the model as potential predictors of CLABSI. Hospital size was defined in the 2017 CDC NHSN model as a binary variable indicating that the number of beds in the hospital was ≥276. 3 Variables were retained using backward selection if they met the significance level of α<0.05.
For both models, we estimated the marginal predicted probabilities of a CLABSI for each patient day in the ICU without including the random effect in the prediction so that hospital characteristics did not influence these values. These predicted probabilities were then used to generate the C statistic and 95% confidence interval (CI) for both models. The C statistic is a measure of discrimination, or the model’s ability to discriminate between those with and without the outcome. The C statistic is the chance that the model will assign a higher probability to patients with CLABSIs than without.Reference Steyerberg 7 Values for the C statistic range from 0.50, a probability no different from chance, to 1.0, which is perfect prediction. Calibration, the model’s ability to accurately quantify the probability of the outcome, was assessed with a calibration plot. The predicted probabilities were plotted against the observed proportion of CLABSI in deciles, and a 45° line was added to visually inspect how well the model was calibrated. In a perfectly calibrated model, the points would rest exactly on the 45° line, implying that the predicted risks are equal to the observed rate.Reference Janssen, Vergouwe, Kalkman, Grobbee and Moons 8 , Reference Crowson, Atkinson and Therneau 9
Unadjusted CLABSI rates were calculated for each hospital by dividing the number of CLABSIs by the total number of ICU days. To calculate risk-adjusted rates, the predicted probabilities from the risk-adjustment model were summed to estimate the expected number of CLABSI events for each hospital. Standardized infection ratios (SIR) for each hospital were calculated by dividing the observed number of CLABSI by the expected number predicted by the ICU-type plus patient case-mix model. An SIR above 1 indicated that the hospital reported a greater number of CLABSIs than expected, while an SIR below 1 indicated that the hospital reported a lower number of CLABSIs than expected by the model. 10 Hospitals were then ranked by the case-mix risk-adjusted SIRs and compared to the rankings when ordered by the ICU-type–only risk-adjusted SIRs.
All analyses were conducted using SAS version 9.4 software (SAS Institute, Cary, NC). The calibration plots were generated using the “ggplot2” package in R studio version 0.99.902 software (R Foundation for Statistical Computing, Vienna, Austria).
RESULTS
In total, 22 hospitals contributed ICU data. The analysis included 85,849 ICU patients, of whom 162 (0.2%) developed CLABSIs. Of the 22 hospitals, 16 (73%) were large (≥296 beds), 11 (50%) were affiliated with medical schools, and 20 (90%) were located in urban areas. Across hospitals, 22,560 (26%) patients were from 9 medical cardiac critical care units, 18,157 (21%) were from 8 medical critical care units, 34,537 (40%) were from 14 medical/surgical critical care units, and 10,595 (12%) were from 6 surgical critical care units based on CDC ICU definitions. All patients had a minimum of 9 ICD-9-CM codes, with a median of 27 and a maximum of 65 codes.
Table 1 presents a bivariate analysis of the relationship between CLABSI and patient demographics and comorbidities. Intensive care unit type, age, coagulopathy, paralysis, liver disease, renal failure, and malnutrition were significant at the P<.10 level in the bivariate analysis. Using the medical cardiac care ICU as the reference category, medical/surgical critical care ICU (P=.06) and surgical critical care ICU (P=.03) were predictive of CLABSI, but the medical critical care ICU (P=.40) was not. Table 2 presents the results of the ICU-type plus patient case-mix model. The following variables were added to the ICU-type–only model: coagulopathy (P=.01), paralysis (P=.03), renal failure (P<.01), malnutrition (P<.01), and patient age in 10-year increments (P<.01). Facility hospital size (P=.33) and medical school affiliation (P=.152) were not significant predictors of CLABSI and were therefore dropped from both models.
TABLE 1 Characteristics of 85,849 Patients With and Without Central-Line–Associated Bloodstream Infection (CLABSI) Admitted to the Intensive Care Unit Between January 1, 2012, and December 31, 2013
NOTE. HIV/AIDS, human immunodeficiency virus/acquired immune deficiency syndrome.
TABLE 2 Hazard Ratios, P Values, and the C Statistic for the ICU-Type Plus Patient Case-Mix Model
NOTE. CI, confidence interval.
The C statistics were 0.55 (95% CI, 0.51–0.59) for the ICU-type–only model and 0.64 (95% CI, 0.60–0.69) for the ICU-type plus patient case-mix model, with a statistically significant difference (P<.001) (Figure 1). When the hospitals were ranked by adjusted SIRs and compared (Table 3), 10 hospitals (45%) changed rank (4 increased in rank and 6 decreased in rank) when comorbidities were added to the ICU-type–only model. Figures 2 and 3 show the calibration of the ICU-type–only model and the ICU-type plus patient case-mix model. Our final model shows better calibration than the ICU-type–only model, which overestimated the expected rate relative to the observed CLABSI rate in some subgroups.
FIGURE 1 Receiver operating characteristic (ROC) curves comparing the intensive care unit (ICU)-type–only model to the ICU-type plus patient case-mix model.
FIGURE 2 Calibration curve for the intensive care unit (ICU)-type–only model.
FIGURE 3 Calibration curve for the intensive care unit (ICU)-type plus patient case-mix model.
TABLE 3 Ranking of HospitalsFootnote a With the Intensive Care Unit (ICU)-Type–Only Model and ICU-Type Plus Patient Case-Mix Risk Adjustment
NOTE. SIR, standardized infection ratio.
a In order of ICU-type-only model ranking.
DISCUSSION
In this retrospective cohort study, we have illustrated the importance of adjusting for patient case-mix variables including comorbid conditions when comparing CLABSI rates across hospitals. Other than the existing CDC model, this analysis is the first in developing risk-adjustment models for CLABSI. Furthermore, the CDC models do not incorporate comorbid conditions or other significant patient factors such as age. Although our model incorporating these factors showed modest discrimination, it showed better discrimination than a model using only ICU type (CDC risk model until 2017). The additional 2017 CDC variables of medical school affiliation and facility hospital size were not statistically significant predictors of CLABSI in our cohort.
We have further demonstrated the importance of risk adjustment by showing the change in rankings of the hospitals that resulted when the risk adjustment model including comorbid conditions was applied. Hospitals with a large burden of patients with more comorbid conditions are expected to have larger CLABSI rates, and their rankings will improve once the risk adjustment model is applied. Likewise, hospitals that serve healthier patients with fewer comorbidities may decline in their performance rankings when SIRs are adjusted for patient case mix. These shifts may have consequences regarding payments and penalties for individual hospitals when all US hospitals are included in this ranking, as is currently done by the CMS.
The CDC models prior to 2017 only adjusted for type of ICU. 10 The new 2017 CDC model added medical school affiliation and facility hospital size as variables. 3 Although these variables are unlikely causally related to CLABSI occurrence, they were probably selected as proxy variables for patient case mix. However, while medical school affiliation may represent a case mix of patients who have more comorbid conditions and higher severity of illness that merits risk adjustment, it may also represent more inexperienced providers that should not be adjusted for when the intent is to use those adjusted rates for quality-of-care comparisons. Similarly, facility hospital size is likely associated with several patient case-mix and care delivery factors, which make the direction of influence on CLABSI difficult to predict. Indeed, in our large and diverse cohort, neither medical school affiliation nor facility hospital size were significantly associated with CLABSI. Therefore, we suggest that it is better to directly adjust for patient demographics and comorbid conditions when possible.
Our analysis has several strengths. Infection preventionists used standardized CDC NHSN criteria to identify CLABSI such that outcome assessment is comparable across hospitals. We used comorbid conditions from discharge codes already collected routinely for other purposes; therefore, the incorporation of these variables into current national risk adjustment would not require any additional data collection burden on the part of hospitals. In fact, ICD diagnostic codes are already routinely transmitted to CMS by hospitals. The use of discharge codes may also encourage the use of risk adjustment because ICD diagnostic codes are easier to access and are collected on every patient by trained individuals in a standardized fashion.
Our approach has some limitations. Most of our sample consisted of large, urban facilities, which may limit the generalizability of our findings to other hospitals. The Premier database did not have data on central-line days, so we were unable to use this measure for our denominator or to account for patients with >1 line. Our use of ICU days as the denominator may have underestimated the overall CLABSI rate in each unit, which may have misclassified patient time at risk, but we have no reason to believe that this misclassification is differential. Work by Horstman et alReference Horstman, Li, Almenoff, Freyberg and Trautner 11 has shown that ICU days correlate strongly with device days and that hospital performance rankings using either measure are also strongly correlated. A criticism of the use of ICD-9-CM codes in research is that they fail to capture all patient comorbidities and could reflect codes that maximize reimbursement.Reference Quan, Parsons and Ghali 12 , Reference Schneeweiss and Maclure 13 Research comparing the Charlson and Elixhauser comorbidity indices derived from ICD-9-CM codes to those same scores extracted from chart review revealed that the sensitivity of the individual components varies greatly but that specificity is nearly 100%.Reference Quan, Li and Saunders 14 , Reference Leal and Laupland 15 Therefore, while some patient comorbidities may have been missed due to low sensitivity of the ICD codes, a condition assigned to a patient is likely to be correct.Reference Quan, Li and Saunders 14 – Reference Needleman, Buerhaus, Vanderboom and Harris 16 Therefore, we may have underestimated the prevalence of these conditions in our study, resulting in smaller rank changes after adjustment. Despite this limitation, our models still demonstrated good discrimination. Another limitation is that we used ICD-9-CM codes and hospitals have recently switched to ICD-10 codes; however, this change is unlikely to affect the discrimination of our model because the identified comorbid conditions can be directly compared between ICD-9-CM and ICD-10. 17
Our analyses demonstrate the importance of using individual demographic data and comorbidities in risk-adjustment models. We believe that the CDC and CMS should strongly consider incorporating comorbid conditions obtained by electronically available ICD codes into their risk adjustment models for CLABSI.
ACKNOWLEDGMENTS
This project was partially supported by the Agency for Healthcare Research and Quality (AHRQ; grant no. R01HS022291), the National Institute of Allergy and Infectious Diseases (NIAID; grant no. 2K24AI079040-06), and the US Department of Health and Human Services. The opinions expressed here are those of the authors and do not reflect the official position of AHRQ, NIAID, or the US Department of Health and Human Services. T.J.L. is employed by Premier, Inc. All other authors have no other conflicts of interest to disclose.
