Surgical site infections (SSIs) are among the most common healthcare-associated infections (HAIs) worldwideReference Lewis, Moehring, Chen, Sexton and Anderson 1 , Reference Magill, Edwards and Bamberg 2 ; they are associated with significant morbidity, mortality, and cost.Reference Zimlichman, Henderson and Tamir 3 , Reference Klevens, Edwards and Richards 4 Surveillance of SSIs has been established as an important preventive measure,Reference Brandt, Sohr, Behnke, Daschner, Rüden and Gastmeier 5 , Reference Anderson, Podgorny and Berríos-Torres 6 and the need for accurate, timely, and efficient methods for detecting SSIs is becoming more evident. However, conventional surveillance methods based on manual chart review are labor intensive and prone to interobserver variability.Reference Calderwood, Ma and Khan 7 Recently, electronic surveillance systems have been increasingly utilized for the surveillance of HAIs.Reference Grota, Stone, Jordan, Pogorzelska and Larson 8 – Reference Sips, Bonten and van Mourik 10 Such semiautomated electronic surveillance systems have been reported to reduce the workload for chart reviews while maintaining a high sensitivity of SSI detection for specific surgical procedures such as total hip and knee arthroplasty.Reference Sips, Bonten and van Mourik 11 – Reference Inacio, Paxton and Chen 15 However, previous studies have been limited to certain types of surgery. The goal of this study was to verify the validity of semiautomated SSI surveillance using electronic screening algorithms in 38 categories of surgery.
Methods
Study setting and design
This study was conducted at the Samsung Medical Center, a 1,989-bed tertiary-care referral center in Seoul, Republic of Korea, in which ~45,000 surgical procedures are performed every year. All surgical procedures in the 38 categories monitored under the conventional SSI surveillance system between January 1, 2013, and December 31, 2014, were included in this study. The surveillance dataset stored in the conventional SSI surveillance registry was used as the reference standard for validating the semiautomated SSI surveillance system. All SSI surveillance was performed based on decisions by the hospital’s Infection Control Committee. The semiautomated surveillance system was developed and implemented for quality improvement.
Conventional SSI surveillance
Since 2013, for the quality improvement of in-hospital patient care, prospective SSI surveillance has been implemented for 38 categories of surgery. With the exception of pacemaker surgery, these categories are consistent with the National Healthcare Safety Network (NHSN) operative procedure categories. 16 The categories are shown in Supplementary Table 1. The conventional SSI surveillance was based on surgeons’ voluntary self-reporting using the electronic program and a comprehensive audit of all the cases performed by the infection preventionists (IPs). Mean rates of voluntary reporting for the 38 categories of surgery during the study period were 16·7% in 2013 and 19·6% in 2014. To detect cases of SSI, IPs reviewed the electronic medical records of all patients who underwent target surgical procedures during the study period, including physicians’ progress notes, nursing records, microbiology and radiology reports, antimicrobials administered, reoperation records and discharge summaries. The medical records of readmissions, emergency visits, and outpatient visits after discharge were also reviewed. The CDC NHSN criteria for SSIs were used, and SSIs were classified as superficial incisional, deep incisional, or organ/space SSIs. 16 The surveillance periods after surgery were 30 days for surgeries without prosthetic implants and 90 days for surgeries with prosthetic implants. Surveillance data were collected prospectively on all patients during the surveillance period. For each case of SSI identified by the IP audit, queries were sent to the surgeon responsible for the operation and further information was collected. However, the surgeons did not have a chance to override the decision based on the NHSN criteria.
Semiautomated SSI surveillance using electronic screening algorithms
To reduce the chart review workload of the IPs, electronic screening algorithms were developed. The algorithms for detection of probable SSI events comprised 3 criteria: (1) antibiotics were ordered after postoperative day 5; (2) microbial cultures were done; and (3) an inpatient infectious disease (ID) specialist consultation was requested during the postoperative surveillance period. To exclude prolonged administration of antibiotics for prophylactic purposes, only the cases in which antibiotics were administered 5 days after surgery were included in the criteria. For the microbial culture criteria, the specimens obtained at the time of the surgery were excluded. All data used by screening algorithms were automatically extracted from a hospital information system. Cases that met at least 1 of the criteria were flagged in the SSI Surveillance Registry and were subsequently reviewed by the IPs. Since July 2016, this semiautomated SSI surveillance system has been in place for 38 categories of surgery.
Validation of the semiautomated SSI surveillance system
The dataset for all surgical procedures under conventional SSI surveillance between January 2013 and December 2014, stored in the SSI Surveillance Registry, was used to verify the validity of the semiautomated SSI surveillance system. The conventional SSI surveillance was used as a reference for the calculation of sensitivity, positive predictive value (PPV), and specificity of the new surveillance method. The validity of the semiautomated SSI surveillance system was analyzed according to wound class, type of SSI, and SSI risk index.Reference Gaynes, Culver, Horan, Edwards, Richards and Tolson 17 The impact of the new surveillance method on workload was determined by counting the number of flagged cases requiring chart review by the IPs and by calculating the estimated time spent on chart review for all flagged cases. The estimated time spent on chart review for each case of surgical procedure requiring a 30-day postoperative surveillance period was calculated by applying the following criteria based on the actual experiences of the IPs: (1) 15 minutes for SSI cases in clean or clean-contaminated surgery; (2) 3 minutes for non-SSI cases in clean or clean-contaminated surgery; (3) 25 minutes for SSI cases in contaminated or dirty-contaminated surgery; and (4) 5 minutes for non-SSI cases in contaminated or dirty-contaminated surgery. For surgical procedures requiring a 90-day postoperative surveillance period, the estimated time required for chart review was calculated at twice the aforementioned values.
Results
Among the 40,516 surgical procedures, a total of 575 SSI events (1·42%) were identified by conventional SSI surveillance. The SSI rate was 0·67% (131 of 19,672) among clean surgeries; the SSI rate was 2·09% (433 of 20,736) among clean-contaminated surgeries; and the SSI rate was 10·19% (11 of 108) among contaminated or dirty-contaminated surgeries. Using the electronic screening algorithms, 13,511 of the 40,516 surgeries (33·3%) were flagged for chart review by the IPs, and 556 (1·37%) were confirmed as SSI cases (Table 1). The probable SSI cases flagged by the algorithms included 5,599 cases (28·5%) among clean surgeries, 7,820 cases (37·7%) among clean-contaminated surgeries, and 92 cases (85·2%) among contaminated or dirty-contaminated surgeries. The sensitivity of the semiautomated SSI surveillance system was 96·7% (556 of 575 cases). The sensitivity of SSI detection was 98·5% for clean surgeries; the sensitivity of SSI detection was 96·1% for clean-contaminated surgeries; and the sensitivity of SSI detection was 100% for contaminated or dirty-contaminated surgeries. The sensitivity, specificity, and PPV for each procedure are shown in Supplementary Table 1. The SSI cases that had not been identified by the electronic screening algorithms included 17 cases of superficial incisional SSI among clean-contaminated surgeries and 2 cases of superficial incisional SSI among clean surgeries. In the 17 unidentified cases of SSI, treatment depended only on wound dressing, without using antibiotics. In addition, 2 other cases of SSI were diagnosed at other hospitals after discharge.
Table 1 Detection Sensitivity (Classified by SSI Type) Using Semiautomated Surveillance
NOTE. SSI, surgical site infection; CI, confidence interval.
The first criterion (antibiotic prescription) used by the screening algorithms flagged a total of 10,910 cases, including 516 SSI cases. The second criterion (microbial culture) flagged an additional 2,474 cases, including 40 SSI cases. The third criterion (consultation with an ID specialist) flagged an additional 127 cases but no SSI cases (Fig. 1). The performance of each criterion for the detection of probable SSI events is shown in Table 2. The SSI detection sensitivity of antibiotic prescription was 89·7% (516 of 575 cases); the SSI detection sensitivity of microbial culture was 89·0% (512 of 575 cases); and the SSI detection sensitivity of ID specialist consultation was 53·7% (309 of 575 cases). Using screening algorithms, the PPV was 4·1%. The third criterion showed the highest PPV (15·1%) among the 3 criteria.
Fig. 1 Venn diagram showing the number of surgical cases flagged using each criterion of the electronic screening algorithms. NOTE. SSI, surgical site infection; ID, infectious disease.
Table 2 Screening Criteria Performance in Detecting Probable Cases of SSI
NOTE. SSI, surgical site infection; PPV, positive predictive value; ID, infectious disease.
The cases flagged by the electronic screening algorithms included 16 of 17 surgical cases (94·1%) with a risk index of 3, 833 of 1,108 (75·2%) cases with a risk index of 2; 5,626 of 10,262 (54·8%) cases with a risk index of 1; and 7,036 of 29,129 (24·2%) cases with a risk index of 0 (Table 3). The SSI detection sensitivity was 100% for surgery with a risk index of 2 or 3, 98·8% for surgery with a risk index of 1, and 94·4% for surgery with a risk index of 0.
Table 3 Number of Flagged Cases and Sensitivity of SSI Detection Using Semiautomated Surveillance, According to the NHSN Risk Index
NOTE. NHSN, National Healthcare Safety Network; SSI, surgical site infection; CI, confidence interval.
Semiautomated SSI surveillance using electronic screening algorithms reduced the number of cases for chart review from 20,258 to 6,756 per year (a 66·7% decrease) and reduced the chart review workload of the IPs from 1,283 to 482 person hours per year (a 62·4% decrease).
Discussion
In this study, we used a large, conventional SSI surveillance dataset with 38 categories of surgery to validate a semiautomated SSI surveillance system based on electronic screening algorithms. We confirmed that the semiautomated system can detect SSI cases with high sensitivity while significantly reducing the workload of IPs. In contrast to previous studies that only covered limited types of surgery,Reference Sips, Bonten and van Mourik 11 – Reference Inacio, Paxton and Chen 15 our study included 38 categories of surgery. Validation of the semiautomated surveillance system was possible because of the large, conventional SSI surveillance database that had already been built, which was used as a reference standard. That dataset includes SSI surveillance data for 40,516 surgical operations in 38 NHSN categories, which accounts for ~50% of all the surgeries performed during the study period. Current conventional surveillance methods, which are dependent on chart review, are so resource intensive that most hospitals monitor only select surgical procedures. Our study shows that a semiautomated surveillance system using electronic screening algorithms and select criteria (ie, antibiotic prescription, microbial culture, and ID specialist consultation) can detect SSI events across the various categories of surgery. We also verified the validity of the semiautomated SSI surveillance system by analyzing cases according to wound class, type of SSI, and SSI risk index.
A systematic review on the impact of electronic surveillance system on IP resources demonstrated a reduction in IP staff time to undertake surveillance in 13 studies ranging from 12·5% to 98·4%.Reference Russo, Shaban, Macbeth, Carter and Mitchell 18 Our results are consistent with previous studies demonstrating that physician requests for wound cultures or antibiotic prescriptions are useful for monitoring SSIs electronically.Reference Branch-Elliman, Strymish, Itani and Gupta 19 – Reference Spolaore, Pellizzer and Fedeli 21 The 3 criteria used in our study produced a high sensitivity of 96·7%, although the specificity was low (67·6%). In validation studies using electronic surveillance methods for SSI, sensitivity ranged from 60·0% to 97·8%.Reference Freeman, Moore, García Álvarez, Charlett and Holmes 22 A systematic review also demonstrated that recent electronic HAI detection systems showed a bias toward higher sensitivity at the expense of specificity.Reference Jeroen 23 The PPV in our study was very low when SSI surveillance depended solely on screening algorithms without the manual chart review of flagged cases. When using antibiotic prescription as a criterion, false-positive cases may include those in which antibiotics are administered to treat infections other than SSI (eg, postoperative pneumonia or urinary tract infection) or those in which prophylactic antibiotics are given for a prolonged duration. The criterion of microbial culture may also flag false-positive cases, including those in which microbial culture is performed for the etiologic diagnosis of an infection other than SSI. The criterion of consultation with an ID specialist provided a lower SSI detection sensitivity than the other criteria, but provided the highest PPV. This might be because surgeons tend to request consultation with an ID specialist only for severe cases of SSI. Some previous studies have focused on the use of billing codes to help improve SSI surveillance, although they were limited to certain types of surgery.Reference Bolon, Hooper and Stevenson 12 , Reference Inacio, Paxton and Chen 15 However, a Korean study regarding feasibility of using administrative data to identify HAI showed that the rates of HAI was significantly underestimated compared with results of medical record survey.Reference Kim, Hwang, Park, Chae and Choi 24 Therefore, billing codes were not considered in this study.
An analysis of SSI detection by the electronic screening algorithms showed that 9·4% of the superficial SSI cases could not be detected by the algorithms. This finding can be explained in 2 ways. First, some patients with mild superficial SSI are managed without antibiotics or wound culture, and these cases can be missed by the current screening algorithms. Second, the diagnosis of superficial SSI is often delayed or missed when its signs and symptoms occur after hospital discharge, and patients are given antibiotics at clinics close to their homes. Clearly, the current algorithms must be complemented to avoid missing such SSI cases. A surgeon’s voluntary reporting can complement such limitations. In fact, 9 of the 19 SSI events that could not be flagged by the screening algorithms had been voluntarily reported by surgeons during the conventional SSI surveillance period.
A major challenge in developing fully automated SSI surveillance methods involves the conflict between increasing sensitivity and increasing the PPV. It may be possible to develop screening algorithms specific for each type of surgery. Ultimately, the use of artificial intelligence technologies (eg, natural language processing and machine learning) could be a promising solution in developing fully automated SSI surveillance systems.
This study had several limitations. First, the dataset used as a reference standard originated from conventional SSI surveillance techniques, which are largely dependent on chart reviews by IPs. As such, it is possible that some cases of SSI went undetected. In particular, identification of superficial SSIs developing after discharge may have affected the validation of our study. Second, our results are from a single center and may not be generalizable to all hospitals. Third, our semiautomated SSI surveillance system is limited in that the PPV generated by the algorithms was low, and all flagged cases must still be reviewed by IPs. In particular, 75% of surgeries with a risk index of 2 or 3 still needed chart reviews. Nevertheless, it improved the efficiency of the IPs by significantly reducing workload while providing a high SSI detection sensitivity.
In conclusion, our semiautomated surveillance system with electronic screening algorithms, together with chart reviews of selected cases, provided high-validity surveillance results. The semiautomated surveillance method significantly reduced the workload of the IPs compared to the conventional surveillance method.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/ice.2018.116
Financial support
No financial support was provided relevant to this article.
Conflicts of interest
All authors report no conflicts of interest relevant to this article.
