Aimed at facilitating real-time surveillance and early intervention, prediction models for healthcare-associated infections (HAIs) have proliferated as healthcare data have become increasingly accessible and robust. Reference Wiens and Shenoy1–Reference Beeler, Dbeibo and Kelley4 Though model performance has improved with new data sources and engineering methods, actionable models that provide individualized infection probabilities that are updated continuously based on patients’ actual hospital event sequences and care trajectories are lacking. Reference Thorsen-Meyer, Nielsen and Nielsen5
In addition, nurse staffing adequacy, a key predictor of infection risk, Reference Shang, Needleman, Liu, Larson and Stone6,Reference Mitchell, Gardner, Stone, Hall and Pogorzelska-Maziarz7 has been largely omitted from prediction models due to challenges with defining and capturing this factor in electronic records. Reference Wiens, Guttag and Horvitz8–Reference Sánchez-Hernández, Ballesteros-Herráez, Kraiem, Sánchez-Barba and Moreno-García12 Staffing adequacy represents the balance between staffing supply (the number and composition of staff on a unit) and patient demand (the type and quantity of care required by patients on a unit). Reference Kalisch, Friese, Choi and Rochman13,Reference Jackson, Chiarello, Gaynes and Gerberding14 Although staffing supply can be measured and extracted from electronic records, existing methods of determining patient demand using healthcare data have significant limitations. Reference Shang, Stone and Larson15,Reference Morris, MacNeela, Scott, Treacy and Hyde16 To more accurately capture patient demand, we previously developed and validated the Nursing Intensity of Care Index to quantify patients’ daily nursing care needs based on their actual use of services. Reference Larson, Cohen, Liu, Zachariah, Yao and Shang17
This study had 2 main objectives. The first objective was to develop and validate a simulation model to estimate patients’ daily probabilities of seven outcomes (bloodstream infection, urinary tract infection, surgical site infection, Clostridioides difficile infection, pneumonia, length of stay, mortality) based on time-varying patient- and unit-level factors, including nurse staffing adequacy. The second objective was to examine whether the incidence of HAIs considered to be preventable through evidence-based nursing care (central line-associated bloodstream infection, catheter-associated urinary tract infection, C. difficile infection, pneumonia) Reference Khanafer, Voirin, Barbut, Kuijper and Vanhems18–Reference Pássaro, Harbarth and Landelle22 vary according to nurse staffing adequacy.
Methods
Sample and setting
This federally funded study was performed using data from 2 tertiary- and quaternary-care hospitals, a pediatric acute care hospital, and a community hospital within a single New York City healthcare network. All inpatients discharged from 2012 through 2016 were included (N=562,435). The participating medical centers’ institutional review boards reviewed and approved the study.
The Nursing Intensity of Care Index
The Nursing Intensity of Care Index was developed at the study institution using previously described methods. Reference Larson, Cohen, Liu, Zachariah, Yao and Shang17 In brief, a team of clinician researchers reviewed lists of all procedures occurring at the study institutions and identified those thought to increase workload for nurses on inpatient units. Eleven staff nurses in a range of pediatric and adult units reviewed the curated list, indicated whether the procedures increased their workload by at least 15 minutes per shift, and suggested additional procedures that had not been included. Each procedure in the final list was ascribed a weight based on previously published data describing the time burden of each activity. Reference De Cordova, Lucero, Hyun, Quinlan, Price and Stone23 The elements of the final index, including weights and data sources, are presented in Table 1.
Table 1. The Nursing Intensity of Care Index Variables and Scoring a
Note. HR, human resources; ICD-9/10-CM, International Classification of Diseases 9th or 10th Revision, Clinical Modification; MAR, medication administration record; NORA, Nosocomial Outbreak Reporting Application; POA, present on admission.
a The Nursing Intensity of Care Index is calculated daily for each patient in a unit and averaged to create a daily score for each unit.
b Scoring of medications was determined by tertile of total number administered: 0=none, 1=first tertile, 2=second tertile, 3=third tertile.
Data sources and definitions
Our team of clinician researchers identified fixed and time-varying patient- and unit-level factors that contribute to infection risk and are obtainable from electronic hospital records. Data on these factors were obtained from administrative records, human resources staffing records, medication administration records, perioperative records, provider order entries, structured nursing documentation in the electronic medical record, International Classification of Diseases Ninth or Tenth Revision, Clinical Modification (ICD-9/10-CM) procedure and diagnosis codes, and records of unit-level infectious disease outbreak periods reported to New York State through the Nosocomial Outbreak Reporting Application. 24 The variables included in the simulation model are detailed in Table 2.
Table 2. Patient and Unit Characteristics Included in the Simulation Model
Note. HR, human resources; ICD-9/10-CM, International Classification of Diseases 9th or 10th Revision, Clinical Modification; MAR, medication administration record; POA, present on admission.
HAIs were identified using previously validated case-detection algorithms based on Centers for Disease Control and Prevention National Healthcare Safety Network definitions, and time-stamped records from the institutions’ clinical microbiology laboratories and patients’ ICD-9/10-CM codes. Reference Apte, Neidell, Furuya, Caplan, Glied and Larson25 Bloodstream infection was defined as a positive blood culture with any organism in the absence of a positive culture with the same organism from another body site within the previous 14 days. Urinary tract infection was defined as a positive urine culture with any organism (≥105 CFU/mL or 103–105 CFU/mL plus pyuria). Surgical site infection was defined as a positive wound culture with any organism within 30 days following a National Healthcare Safety Network designated procedure. 26 C. difficile infection was defined by positive stool culture. Pneumonia was defined as a positive respiratory culture with any organism plus any ICD-9/10-CM pneumonia code. Any infection that occurred >2 calendar days after admission was considered healthcare-associated and was included in the study. Lengths of stay and mortality outcomes were obtained from hospital administrative records.
Simulation modeling
We used a non-Markovian simulation to estimate daily conditional probabilities of bloodstream infection, urinary tract infection, surgical site infection, C. difficile infection, pneumonia, length of stay, and in-hospital mortality. In brief, each patient characteristic, unit characteristic, and hospital event in Table 2 was assigned a unique integer. A patient’s current state (X
t
) is a value determined by the presence and order of onset of these integers, representing all that the patient has experienced to date during the admission. The model
$(X_{\{t+1\}}\comma \ h _{\{t+1\}}) =f \{(X_t\comma h_t)\comma \theta\comma u_{\{t+1\}}\}$
outputs the patient’s next state (X
t+1) and an updated memory of previous states (h
t+1) as a function of the patient’s current state (X
t
), the previous memory (h
t
), model parameters θ, and uniform random variable u
t+1. A detailed description of the model is included in Appendix 1 (online), where the model is parameterized using recurrent neural networks.
Model validation
To validate the model, we took an independent random sample of 100,000 patient admissions and simulated a path to discharge (or death) for each. We then compared the average mortality, infection, and length of stay implied by the simulation model with the observed metrics to see how closely the output of the simulation resembles real events. To measure the predictive performance of the model, we devised a comparison with logistic regression using the patient and unit variables available at admission to predict C. difficile infection, noting that this is meant to establish the soundness of the model, as our model formulation can accommodate more complex scenarios than traditional supervised learning predictive models. C. difficile was chosen for the validation because the factors that affect risk are generalized and not heavily dependent on specific events such as surgery or indwelling device placement. We compared model performance using area under the receiver operating characteristic curve (AUC of ROC) as a metric. To illustrate the potential clinical utility of the model, we present case studies illustrating daily changes in outcome probability based on patient and unit characteristics and events.
Nurse staffing adequacy
To evaluate whether incidence of central-line–associated bloodstream infection, catheter-associated urinary tract infection, C. difficile infection, or pneumonia vary according to nurse staffing adequacy, we calculated infection incidence by unit-level nurse staffing (total registered nurse, licensed practical nurse, and nursing assistant hours per patient day by tertile) and unit-level patient care intensity (Nursing Intensity of Care Index by tertile) on the day of infection.
Results
Model performance
Table 3 presents performance metrics for the simulation model. Most metrics implied by the model fall within 95% confidence intervals of the actual estimates (±1.96σn), meaning that the simulation model captures the structure and order of patient transitions from admission to discharge. The predictive power of our simulation model was slightly higher than that of the logistic regression model (AUC 0.852 and 0.816, respectively) (Fig. 1). The case studies presented in Figure 2 illustrate how our simulation model can provide insights beyond traditional supervised learning methods by accounting for the timing of each additional risk bearing event as well as the history and sequence of previous risk bearing events to calculate a more individualized real-time assessment of the probability of infection.
Table 3. Key Metrics Implied by the Simulation Model Compared to Reality After Simulating 100,000 Patient Paths
Figure 1. Baseline comparison between simulation model and logistic regression using only variables available at admission to predict Clostridioides difficile.
Figure 2. In Case Study A, a 13-year-old female with a malignancy was admitted with a low Pediatric Chronic Complex Condition score (0-6) to a unit with higher staffing (upper tertile) and moderate Nursing Intensity of Care unit score (middle tertile). She developed Clostridioides difficile infection (CDI) on day six after taking antibiotics for six consecutive days. Her daily risk of C. difficile infection estimated from the simulation model is plotted and annotated with landmark risk factors, illustrating how the probability of infection progressively increases with each consecutive day of antibiotics and following transfer to the intensive care unit (ICU). This case study highlights the usefulness of the simulation model to jointly assess the risk of an outcome that is caused by multiple co-dependent time-varying factors occurring simultaneously, which is difficult to achieve with traditional supervised machine learning methods. In Case Study B, a 62-year-old female diabetic patient with a malignancy was admitted with a high Charlson Comorbidity score (7-13) to a unit with lower staffing (lower tertile) and higher Nursing Intensity of Care unit score (upper tertile). She developed a urinary tract infection (UTI) on day 23 after seven days with a urinary catheter. Her daily risk of urinary tract infection is plotted, which illustrates how the probability of infection progressively increases after an initial spike following antibiotic administration beginning on day seven, urinary catheter insertion on day 16, and the addition of new IV push medications on day 20.
Nurse staffing adequacy and infection
Table 4 presents a matrix showing the percent of patients with central-line–associated bloodstream infection, catheter-associated urinary tract infection, C. difficile infection, and pneumonia under a range of nurse staffing and nursing care intensity conditions. For all 4 types of infection, incidence was greatest when staffing was lowest and nursing care intensity was highest.
Table 4. Infection Incidence by Level of Nurse Staffing and Care Intensity
Data are infection incidence per 100 patients (standard error). The highest infection incidence within the matrix for each infection type is shown in bold.
Discussion
Prediction models for identifying preventable hospital-acquired conditions, such as HAIs, have great promise for improving patient outcomes. Reference Wiens and Shenoy1 Still, the utility of prediction models in patient care remains limited due to low predictive power, lack of timely, actionable interventions, or inability to capture ongoing changes in patient risk. Reference Scardoni, Balzarini, Signorelli, Cabitza and Odone27,Reference Luz, Vollmer, Decruyenaere, Nijsten, Glasner and Sinha28 In this study we addressed 2 major limitations of HAI prediction models. First, we used a novel approach to modeling the likelihood of infection, death, and length of stay by estimating daily probabilities that are conditional not only on the events that the patient experienced to date during their hospital stay but also the sequence of those events. Second, we used a novel approach to measuring nurse staffing by considering both staffing supply and patient care demand using the Nursing Intensity of Care Index. Reference Larson, Cohen, Liu, Zachariah, Yao and Shang17
To our knowledge, this is the first study to incorporate staffing adequacy into a prediction model for HAIs. Countless studies have examined the association between nurse staffing and preventable complications, including HAIs. Reference Aiken, Sloane and Bruyneel29,Reference Schreuders, Bremner, Geelhoed and Finn30 The evidence strongly suggests that staffing plays some role in infection risk, though findings are not consistent across studies. A review by Shang et al Reference Shang, Stone and Larson15 reported that inconsistent findings are likely explained by differences in the type and quality of staffing supply data sources. Our findings suggest that mixed results may also be due to whether and how studies account for patient care demand.
Despite challenges obtaining accurate staffing supply data, definitions of staffing levels and skill mix are relatively consistent among high-quality studies, with nurse-to-patient and registered nurse (RN)-to-support staff ratios among the most common metrics. Reference Shang, Stone and Larson15 Patient care demand, on the other hand, is more difficult to define and measure, and meaningful methods for capturing patient care demand are not incorporated in most studies. Reference Shang, Stone and Larson15,Reference Morris, MacNeela, Scott, Treacy and Hyde16,Reference van Oostveen and Ubbink31 Severity of illness indices designed for mortality risk adjustment are sometimes used as proxies for how much nursing time patients require, but comorbidity burden does not necessarily correlate with inpatient nursing need. Reference van Oostveen and Ubbink31 Case-mix indices based on demographics, procedures, and comorbidities are used to estimate nursing needs at the unit level, yet such measures are insensitive to changes in care demand throughout a patient’s stay. Reference Garcia32 Patient classification and acuity systems incorporate a more holistic perspective of nursing care needs; however, these are generally designed to categorize patients into broad categories of low, medium, and high need. 33
To improve measurement of nursing care demand, we developed and validated the Nursing Intensity of Care Index to capture patients’ nursing needs on a daily basis to account for the fact that needs change over the course hospitalization and depend on more than medical diagnoses and demographic characteristics. Reference Larson, Cohen, Liu, Zachariah, Yao and Shang17,Reference van Oostveen and Ubbink31 Using this index, we demonstrated that infection incidence was greatest when units had the lowest nurse staffing and highest patient care intensity. There is no clear dose–response relationship between staffing and nursing care intensity aside from the most extreme categories, which may suggest a threshold at which the staffing versus intensity ratio becomes unsafe. This warrants further analysis. Although it was not possible to precisely account for incubation period in this analysis, meaning that infection incubation could have begun before the time of staffing measurement, these findings suggest that the Nursing Intensity of Care Index may be useful tool to aid hospitals in safe staffing allocation.
Our study to improve prediction modeling for HAIs has some limitations. First, the data used to train the model are limited to a single health system. Although we aimed to create a model that can be generalized and operationalized elsewhere by including only electronic health data that is widely available across institutions in the United States, we acknowledge that local methods of data capture, billing code practices, order sets, and EMR structure may vary in ways that affect the predictive value of some variables. In addition, hospitals serving different patient populations or offering a different balance of services may need to incorporate other variables that impact infection risk, nursing care intensity, or both. Finally, the usefulness of prediction models for preventable hospital complications is determined in large part by whether they are integrated into practice such that real-time intervention to change a patient’s infection probability is possible. Although our model identified modifiable factors that impact risk of infection, we did not deploy the model in clinical practice to assess its impact on infection rates in a real-world setting; however, the real-time availability of the data elements used for prediction would make this possible.
In summary, this study built on previously developed methods for predicting infection risk using a modeling approach that considers the order in which risk factors occur and incorporates a holistic consideration of staffing adequacy. The model had high predictive value for determining a patient’s risk of infection on each day of hospitalization, and our results suggest that patients have the greatest risk of infection when unit-level staffing is low and patient care demand is high.
Acknowledgments
Financial support
This work was supported by a grant from the Agency for Healthcare Research and Quality (grant no. R01 HS024915).
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2021.114
