Healthcare-associated infections (HAIs) are associated with extraordinary cost, both in terms of patient outcomes and hospital expenses. A single HAI can cost a hospital between $26,040 and $68,146, and such infections are estimated to cost a 200-bed facility more than $1.7 million per year.Reference Pittet, Hugonnet and Harbarth 1 In addition to these monetary costs, the average length of stay of a patient with an HAI increases by 2.61 days.Reference Erasmus, Brouwer and van Beeck 2 Estimates in trauma patients have shown 1.5–1.9-fold higher odds of mortality among patients with an HAI, and those with sepsis have been found to have nearly 6-fold higher odds of death.Reference Glance, Stone, Mukamel and Dick 3 While these numbers may be subject to some debate, the fact that HAIs exert a significant clinical impact is indisputable. Of the estimated 1.7 million HAIs that occur each year in the United States, almost 100,000 are fatal.Reference Klevens, Edwards and Richards 4
Hand hygiene (HH) is widely believed to be the most effective modifiable factor for the prevention of HAIs.Reference Whitby, Pessoa-Silva and McLaws 5 Despite substantial evidence supporting HH as a cost-effective and beneficial approach to infection prevention and control, HH rates remain alarmingly low at most US hospitals, with compliance averaging a mere 20%–40%.Reference Schneider, Moromisato and Zemetra 6 , Reference Boyce, Chartier and Chraiti 7 A challenge to improving HH performance among healthcare personnel (HCP) is the reliable measurement of adherence to these practices. An emerging approach to HH improvement is the adoption and testing of new methods for measuring HH performance. Presently, the gold standard for measuring HH compliance involves the use of trained observers who periodically and covertly assess HCP and record their adherence to accepted HH standards. These standards are often limited to compliance with performing HH upon entering and exiting a patient room (also known as a “wash in, wash out” policy) due in part to physical limitations related to directly observing all recommended opportunities for HH. Direct observation is estimated to capture only 1.2%–3.5% of all HH opportunities in the situations in which it is most rigorously applied.Reference Fries, Tolentino, Thomas, Herman, Segre and Polgreen 8 In addition, the Hawthorne effect has been well described and quantified, with estimates of a 3-fold inflation in HH performance when auditors are present.Reference Eckmanns, Bessert, Behnke, Gastmeier and Ruden 9 , Reference Srigley, Furness, Baker and Gardam 10 This method is resource-intensive and is generally too costly to be applied on a large scale. Alternative approaches have also been inadequate: self-reported behavior assessments are biased, and surrogate markers, such as measurement of hand gel consumption, are unreliable.Reference Haas and Larson 11 , Reference Boyce 12 A reliable method to quantify adherence to HH recommendations will provide the opportunity to ask sophisticated questions about the factors driving HCP behavior and will offer a tool for testing interventions. Thus, such a method will yield valuable data that can quantify the risk of HAI attributable to HH compliance failures.
Hand hygiene monitoring technology (HHMT) using radiofrequency identification (RFID) and infrared technology (IR) to measure the use of soap and alcohol-based hand rub (ABHR) as well as monitoring of the movement of persons throughout a hospital unit has become available on the commercial market. Despite significant advances in technological methods for HH surveillance, a systematic review of 42 articles mentioning automated measuring systems found that <20% of these studies included calculations to determine the accuracy of these systems.Reference Ward, Schweizer, Polgreen, Gupta, Reisinger and Perencevich 13 Without rigorous validation of HHMTs, the implementation of these systems is premature.Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 14 We set out to validate an HHMT (GOJO SMARTLINK, GOJO Industries, Akron, OH) designed to capture HH behaviors at the hospital-unit level.
METHODS
Setting
The University of Chicago Medicine is a major teaching hospital located in Chicago, Illinois. It serves as the primary nexus of clinical care for the south side of Chicago, as the principal teaching hospital for the University of Chicago’s Pritzker School of Medicine, and as a regional referral center for cancer, inflammatory bowel disease, specialty surgery, and many other complex clinical conditions. With an inpatient capacity of 680 beds, the health system facilitates >20,000 hospital admissions and almost 500,000 outpatient visits annually. The medical center provides a full spectrum of care from primary care through tertiary and quaternary care, including transplant and a free-standing pediatric hospital. This study was deemed a quality improvement initiative and not human subjects research and was therefore not reviewed by the institutional review board.
Hand Hygiene Monitoring Technology
The GOJO SMARTLINK electronic monitoring system is comprised of 5 main components: activity counters, dispenser actuation counters, data receivers, a secure server, and a digital monitor. Activity counters are mounted near the doorway in each patient room. These devices are comprised of 2 “detection zones,” invisible cones that monitor thermal infrared energy (ie, heat). A room entry or exit is captured when a human body walks through both detection zones, displacing heat in the zone. Infrared energy has been applied in other settings and is highly reliable at detecting human presence. Incorporation of 2 zones within each activity counter allows for a level of internal validation intended to prevent a room entry or exit from being captured when (1) a person walking in the hallway passes by the doorway or (2) a person in the patient room walks near the doorway without exiting. No wearable devices are needed for the operation of this system.
Dispenser actuation counters are inserted into all ABHR and soap dispensers. These counters are unique housing systems that sit inside the dispensers, resulting in no visible indications of monitoring. Counting mechanisms have been deployed in hospitals throughout the United States and have proven to be accurate in their ability to detect the number of times a dispenser is actuated.
Data receivers are installed throughout the hospital to capture both “pings” confirming the device is online and data counts from each activity counter and dispenser. All data captured (ie, each room entry, room exit, soap dispenser actuation, ABHR dispenser actuation) are time-stamped based on the minute and second the data are received by a receiver. A secure cloud-based server captures all data from the receivers and stores the information at the device level. An online user interface allows for secure login during which all monitored data can be reviewed in tabular or graphical format. This system allows for assessment of data by hospital building, hospital floor, hospital unit, and time.
Finally, a digital monitor is installed at the nurses’ station of each hospital unit, displaying real-time compliance data. Lag time is minimal from the time of activity to the time of display on the digital monitor because the data are sent from each device through a receiver to the cloud-based server. This lag time is often only a few minutes. Compliance data can be displayed in a variety of formats, including tables and trends. Compliance is calculated using a defined period of time as follows:
$${{{\rm No}{\rm .}\,{\rm soap}\,{\rm dispenser}\,{\rm actuations{\plus}No}{\rm .}\,{\rm ABHR}\,{\rm dispenser}\,{\rm actuations}} \over {{\rm No}{\rm .}\,{\rm room}\,{\rm entries{\plus}No}{\rm .}\,{\rm room}\,{\rm exits}}}$$
Data can be evaluated at the hospital floor or building level, but they cannot be accurately assessed at a level of granularity more detailed than the hospital unit. To obtain room-level compliance data, the system would need to assign each soap or ABHR dispenser to a unique room. Then, only HH events performed at dispensers assigned to the room where activity occurred (entry or exit) would be included in compliance percentages. In other words, if an HCP washed their hands in the hallway and then walked into a patient room 2 doors down, the system would not be able to associate the HH event with the room entry. As such, this system is limited to calculating performance by simply adding all HH activities (ie, dispenser activations) and all room activities (ie, entries and exits) during a defined period to provide unit-level compliance. Furthermore, this system cannot distinguish between HCP and visitor activity.
Validation Approach
Our team followed a rigorous validation approach to assess the sensitivity and positive predictive value (PPV) of the HHMT after calibration by the vendor. Each device encounter was treated as an independent event. First, a planned path (ie, a route created throughout a unit or hospital area that allows for systematic validation of HHMT through purposeful activation of each activity counter and HH dispenser) was followed for each hospital unit where the HHMT was installed.Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 14 Thus, system efficacy was assessed by quantifying its ability to capture purposeful behavior of system investigators. The planned path was documented in detail and was compared to the raw data collected by the HHMT. Devices that were unable to accurately capture planned path behavior were adjusted and reassessed, as recommended for the validation of these systems.Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 14
Next, trained observers used direct observation to document room activity on a hospital unit. These observations, collected using the gold standard for measuring HH behavior, were also compared to raw data collected by the HHMT. This behavioral validation phase quantified system efficacy by quantifying its ability to accurately detect real-world behaviors in the hospital or clinic environment.Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 14 Because this particular HHMT cannot distinguish HCP activity from visitor and patient activity, direct observation was used to estimate the proportion of room activity contributed by patients and their families on each unit in an attempt to measure any effect on HHMT accuracy.
A sample size of 123 was deemed necessary to accurately calculate the sensitivity of the HHMT. An α of 0.05 equated to ZX/2=1.96, and sensitivity was conservatively set at 80% based on preliminary observations. The precision or maximum marginal error was set at 10% and prevalence was set at 50%. While prevalence of HH performance is known to range between 20% and 40%, the prevalence for the sample size calculation was an estimate of the proportion of behavior related to HH (eg, entering into a room, exiting a patient room, or actuating a dispenser) of all human movement on the hospital unit. In other words, it was estimated that 50% of movement on the unit was either in the hallway or patient room. This is a conservative estimate based on routine observation of provider behavior, specifically focused on walking through the unit.
Statistical Analysis
Sensitivities, the probabilities that true activity will be captured by the system, were calculated separately for the planned path phase and behavioral validation phase. “True activity” was defined as activity performed by the investigator team during the planned path phase and as activity observed by direct observation during the behavioral validation phase. Similarly, PPVs, the probabilities that activity captured by the system really occurred, were calculated for both planned path and behavioral validation. These analyses facilitated the assessment of system efficacy (ie, accuracy during purposeful activity conducted during the planned path phase) and effectiveness (ie, accuracy during real-world activity documented during behavioral validation). These fundamental epidemiologic measures were then stratified by estimated proportion of visitors and physical location. Analyses were conducted using Excel 2011 (Microsoft, Redmond, WA) and Stata/SE 13.1 for Mac (StataCorp, College Station, TX).
RESULTS
Planned Path
During the planned path phase, system investigators purposefully performed 4,872 unique events across 3 distinct hospital buildings varying in size and age since construction. Overall sensitivity across the medical center was 88.7% with a PPV of 99.2%. System sensitivity was significantly variable across buildings (P<.001) and was higher in newer buildings A (92.6%) and C (93.3%) than in the older building B (85.2%). This trend held when sensitivity was stratified by event type (entry or exit), ABHR dispenser actuation, or soap dispenser actuation (Table 1). While overall PPV did not significantly vary across buildings, stratification across event type varied in PPV for both room entries (P=.046) and room exits (P=.019) among the 3 hospitals (Table 1). Notably, while room entry and exit were assessed as distinct events, the system’s ability to distinguish an entry from an exit when counting events was deemed insufficient.
TABLE 1 Accuracy of HHMT During Planned Path Validation (Purposeful Activation), Stratified by Hospital Building and Event TypeFootnote a
NOTE. HHMT, hand hygiene monitoring technology; PPV, positive predictive value; ABHR, alcohol-based hand rub.
a This unit had observations over the course of changes in device settings. Event type includes room entry, room exit, activation of alcohol-based hand rub (ABHR) dispenser, and activation of soap dispenser.
Overall sensitivity and PPV both varied within buildings and across units. Of particular interest, units facing the southwest direction, which occurred only in building B, had lower rates of sensitivity than comparable units (see Supplemental Materials). We hypothesize that sunlight intensity may have interfered with the ability of the HHMT to detect changes in heat necessary to measure room activity. While results are displayed for all planned path activities conducted, the HHMT devices were adjusted until each unit achieved at least 1 planned path with every device reaching 100% sensitivity on a single activation. On average, devices that failed to measure planned path events were adjusted 1.5 times. These adjustments achieved 100% sensitivity and PPV for both soap and ABHR dispensers. Thus, behavioral validation was limited to room activity (room entries and room exits), which was difficult for the HHMT to capture accurately.
Behavioral Validation
During the behavioral validation phase, trained direct observers recorded 5,539 unique events across 3 distinct hospital buildings (see Supplemental Materials). Overall sensitivity across the medical center was 92.7% and PPV was 84.4%. System sensitivity remained significantly variable across buildings (P=.023) and was again higher in the newer buildings A (94.2%) and C (92.5%) than in the older building B (91.7%). Overall PPV also varied significantly across buildings (P<.001). System sensitivity was slightly higher on inpatient floors (P=.031), while PPV was significantly higher on intensive care unit (ICU) floors (P<.001) (Table 2). Time of day, dichotomized as morning (12:00 a.m.–11:59 a.m.) or evening (12:00 p.m.–11:50 p.m.) did not have a significant impact on HHMT sensitivity (P=.167); however, PPV was more reliable during morning hours (P<.001) (Table 2).
TABLE 2 Accuracy of HHMT During Behavioral Validation (Observation of Natural Behavior), Stratified by Hospital Building, Floor Type, Time of Day, and Proportion of Visitors
NOTE. HHMT, hand hygiene monitoring technology; PPV, positive predictive value; ICU, intensive care unit.
a P values are associated with Student t test or ANOVA as appropriate.
The main confounding factor considered in this analysis was the proportion of room activity contributed by patients and families, collectively referred to as visitors. As seen in Table 2, the proportion of visitors significantly affected system sensitivity (P<.001) and PPV (P<.001). Notably, when the proportion of visitors was divided into quartiles, an inverse dose–response relationship between visitor activity and HHMT PPV was seen. In other words, as the proportion of patient room entries and exits contributed by visitors increased, the probability that room activity captured by the HHMT was a true event decreased.
DISCUSSION
Overall efficacy of the HHMT, as measured by the planned path validation, was high: overall sensitivity was 88.7% and PPV was 99.2%. Assessment of each room entry, room exit, soap dispenser actuation, and ABHR dispenser actuation allowed for targeted readjustment of devices that led to 100% sensitivity and 100% PPV for HH events (ie, soap and ABHR dispenser actuations). Thus, system effectiveness was assessed by measuring the sensitivity and PPV for capturing HH opportunities (room activity). From a technological standpoint, this approach is valid. The ability to accurately measure soap or ABHR dispenser actuations relies solely on a mechanical counter placed in the dispenser. However, the ability to detect heat displacement at a level of accuracy that distinguishes room entry from nearby activity is technologically more complex. Thus, this narrowed scope during the behavioral validation phase was justified.
Effectiveness of the HHMT, as measured by behavioral validation, was similarly high, with overall system sensitivity increasing to 92.7%, while PPV was 84.4%. This decrease in probability that a captured event was an event that actually occurred when compared to the planned path phase was likely due to the exclusion of soap and ABHR dispensers, which, as stated above, reached 100% accuracy. When tested in a natural healthcare environment, the system was highly likely to detect room activity. However, the frequency of false-positive events when measuring room activity is noteworthy: 15.6% of all HH opportunities (entries or exits) were false positives. The most frequently documented events that resulted in false-positive HH opportunities included hovering of persons near the doorway, a series of quick room entries and exits, and room activity accompanied by a mobile computer or piece of medical equipment (ie, ultrasound machine). These false-positive events result in a decreased HH rate as the denominator of HH compliance is artificially increased.
Another notable finding during the behavioral validation phase was the impact of HCP workflows on the performance denominator. For example, placement of medical supplies outside the patient room required nursing staff to frequently enter and exit rooms while carrying supplies. Similarly, terminal cleaning of recently vacated rooms required an estimated 10 room entries and exits on average, per room. This procedure artificially penalizes HH performance rates for units with greater volumes of discharged patients. Finally, the proportion of room activity contributed by patients and families, collectively referred to as visitors, inversely impacted the PPV of the HHMT. We hypothesize that this relationship can be explained by visitor behaviors such as walking throughout the patient room while triggering the room activity counter, hovering near room doorways, and increasing true room activity to accommodate loved ones. In short, visitors exhibit different behaviors than those regularly practiced by HCP in the routines of care.
Objective measures of sensitivity and PPV indicate the promise of the benefit of this and other HHMTs to capture basic behaviors associated with HH. The findings of this validation process support previous recommendations that the accuracy of HHMT should be assessed in each unique physical location given the variation in accuracy detected between buildings, unit type (eg, ICU vs floor), and proportion of visitor activity.Reference Limper, Garcia-Houchins, Slawsky, Hershow and Landon 14
Perhaps the most notable finding of this initiative was the significant impact of HCP workflows and visitor behaviors on HHMT accuracy and thus HH performance. Further technological development is necessary to accurately account for necessary workflows such as transportation of medical equipment into and out of patient rooms, group activity during patient rounding, and visitor presence within patient rooms. However, efforts to redesign workflows around room cleaning, supply storage, and provider communication are likely also necessary to accurately measure HH performance using an aggregate-level HHMT. While adjustment of performance rates to account for system inaccuracies is necessary to accurately inform HCP, utility exists in continuous surveillance to visualize trends in performance.
With the advent of HHMT, questions regarding traditional “wash in/wash out” HH polices are likely to be challenged. Further development of technologies capable of distinguishing behavior near patient rooms or within empty patient rooms may provide a unique opportunity to quantify the risk associated with HH noncompliance across organic hospital workflows to further inform hospital HH policies. Furthermore, the reassessment of this HHMT using the planned path validation approach may be necessary to ensure continued proper calibration across time.
ACKNOWLEDGMENTS
We would like to acknowledge the ongoing contributions of the Hand Hygiene Leadership Committee at the University of Chicago Medicine.
Financial support: No financial support was provided relevant to this article.
Potential conflicts of interest: Emily Landon has been an invited speaker for Proventix and GOJO. Dr. Landon has also provided consulting services for Proventix. All other authors report no conflicts of interest relevant to this article.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2016.298
