Since June 2011, all general acute-care hospitals in California have been required to submit data on surgical-site infections (SSIs) following both colon surgery and abdominal hysterectomy to the Centers for Disease Control and Prevention National Healthcare Safety Network (CDC NHSN). 1 This requirement was extended to all states in 2012 2 ; these data are publicly available on the Medicare Hospital Compare website. 3
Beginning in fiscal year 2016, SSIs following these 2 procedures began to impact financial reimbursement for US acute-care hospitals through both the Hospital Value-Based Purchasing (VBP) Program and the Hospital-Acquired Condition (HAC) Reduction Program. 4 , 5 Under these 2 Centers for Medicare and Medicaid Services (CMS) programs, up to 3% of a hospital’s Medicare revenue is now at risk based on performance compared with other US hospitals. 6 Each hospital’s performance is determined by SSI rates which are self-reported through NHSN. 7
Prior work, however, has raised concerns about variable surveillance practices among hospitals. SSI surveillance has been difficult to standardize because most infections occur post discharge and because hospitals employ a wide array of surveillance methods, such as reliance on reporting by surgeons and other hospital staff, review of postoperative microbiology and readmissions, and chart review of all surgical cases.Reference Avato and Lai 8 – Reference Koek, Willie, Isken, Voss and van Benthem 11 The methods used impact the rates being used for performance comparisons.
Billing codes have been shown to improve SSI case capture by identifying cases missed by traditional surveillance.Reference Huang, Placzek and Livingston 12 – Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe 15 In fact, this “indirect method” of SSI surveillance was included in the expert guidance recommendations of the Society for Healthcare Epidemiology of America.Reference Anderson, Podgorny and Berrios-Torres 16 This method improves SSI detection, and it does so with enhanced efficiency compared to complete review of all performed surgeries.Reference Calderwood, Ma and Khan 13 For this reason, claims-based surveillance is currently being employed by some state health departments along with CMS as a method for validating publicly reported data. 17
In 2013, the California Department of Public Health (CDPH) performed on-site validation to evaluate SSI reporting following colon surgery and abdominal hysterectomy. The goal was to better understand how well SSI surveillance was being performed and to improve standardized capture of SSI across the state. In addition, the validation allowed CDPH the opportunity to provide one-on-one education, to identify common errors, and to develop targeted education for adoption of a standardized SSI surveillance method by all California hospitals.
METHODS
Study Participants
As part of the 2013 CDPH SSI Validation Project, up to 50 hospitals across the state of California were asked to volunteer to have their SSI data validated (convenience sample). This validation involved an on-site 2-day review at each hospital by 1–2 infection preventionists from the CDPH Healthcare-Associated Infections Program.
Each volunteer hospital was asked to provide a list of all colon and abdominal hysterectomy procedures performed over a 1–2-quarter period, based on the same procedure codes used to report in the NHSN. 18 In addition, each hospital was asked to provide all postoperative diagnosis and procedure codes within 30 days of the surgical procedure, including from the surgical admission and from any readmissions to the surgical hospital within the 30-day postoperative window.
If possible (but optionally), hospitals were asked to provide a list of patients who received antibiotics for ≥8 days following colon surgery or ≥4 days following abdominal hysterectomy.Reference Yokoe, Khan and Olsen 14 This procedure was added to test a hypothesis that adding antibiotic surveillance might enhance claims-based surveillance and to capture additional SSI cases.
Site Visit Validation Procedure
During the site visit at each volunteer hospital, the CDPH infection preventionists reviewed the following records for evidence of an SSI using NHSN criteria. 18 , Reference Horan, Andrus and Dudeck 19 First, they reviewed the full-text medical record for patients “flagged” by an International Classification of Diseases, Ninth Revision (ICD-9) diagnosis or procedure code indicative of a possible SSI. This was based on a previously validated code set that has been used nationally by CMS to select records for SSI validation following these 2 procedures.Reference Letourneau, Calderwood, Huang, Bratzler, Ma and Yokoe 15 , 17 Second, the CDPH infection preventionists re-reviewed the full-text medical records for all patients that had been reported to NHSN as having an SSI but who were not “flagged” by an ICD-9 surveillance code.
At the completion of the validation process, the results were reviewed with each hospital, and each hospital was asked to report any missed cases to NHSN and to correct any reported cases that were not confirmed as SSI based on the findings.
Analysis
We compared the sensitivity of surveillance triggered by SSI “flag” codes to that of hospital case identification using traditional surveillance methods. Of the total SSI cases confirmed on validation review, we calculated the number identified by traditional surveillance methods and the number identified by SSI “flag” codes. This proportion is the sensitivity of each method. We also evaluated the specificity of traditional hospital surveillance looking for reported cases that did not meet NHSN surveillance criteria and the positive predictive value (PPV) of surveillance triggered by codes suggestive of a possible SSI to estimate the efficiency of this surveillance methodology.
Comparing cases identified by review triggered by SSI “flag” codes to cases identified by hospital surveillance prior to the validation visit, we compared the proportion of patients in each group with specific NHSN surveillance criteria for SSI, positive microbiology, hospital readmission in the postoperative surveillance window, reoperation at the same site in the postoperative surveillance window, and timing of SSI (during the index admission vs postdischarge). We used the Fisher exact test to evaluate any differences.
For those patients who had been reported prior to the CDPH validation survey but were missed by claims-based surveillance, we evaluated whether these patients would have been identified by antibiotic surveillance as an alternative enhanced surveillance methodology. We also evaluated the added work of this surveillance strategy in terms of the positive predictive value of reviewing cases flagged by antibiotic days alone.
ICD-9 to ICD-10 Crosswalk
Finally, for both procedures, we performed a crosswalk of our validation codes from ICD-9 to ICD-10 because ICD-10 became the standard in October 2015. 20 We included a crosswalk of diagnostic codes only because procedure codes have been found to add little benefit to diagnostic codes and because the ICD-10 procedures have an impractically complex mapping to current ICD-9 procedure codes.Reference Calderwood, Kleinman and Bratzler 21 , Reference Calderwood, Kleinman, Murphy, Platt and Huang 22 Our validation code set for abdominal hysterectomy only included ICD-9 diagnosis codes.
RESULTS
Volunteer Hospitals
In total, 47 California hospitals volunteered to participate in the 2013 CDPH SSI Validation Project: 42 of these hospitals performed colon surgery, and 34 of these hospitals performed abdominal hysterectomy. Table 1 shows the characteristics of the 429 general acute-care hospitals in California overall, including a breakdown of the 47 hospitals that volunteered to have their SSI data validated and the 382 nonparticipating hospitals. On-site validations were performed between March and July 2013.
TABLE 1 Hospital Characteristics
NOTE. LA, Los Angeles.
Surveillance for SSI following Colon Surgery
The left column of Table 2 shows the performance of traditional hospital surveillance versus claims-based surveillance for identifying SSIs following colon surgery. During the on-site validations, the CDPH infection preventionists reviewed full-text medical records for 561 patients who had undergone colon surgery, including 39 patients identified by traditional surveillance only, 82 patients identified by both traditional surveillance and claims-based surveillance, and 440 patients identified by claims-based surveillance only. Overall, the CDPH infection preventionists identified 239 SSIs (82 superficial incisional SSIs, 55 deep incisional SSIs, and 102 organ/space SSIs). Of these 239 SSIs, 159 (67%) occurred during the same admission as the surgical procedure and 80 (33%) were identified post discharge.
TABLE 2 Comparative Performance of Traditional Hospital Surveillance Versus Claims-Based Surveillance
NOTE. SSI, surgical-site infection; NHSN, National Healthcare Safety Network.
a Unless specified as percentage.
b Selected for review based upon presence of a claims code suggestive of SSI
Traditional hospital surveillance performed prior to the CDPH validation visit had identified only 120 of the 239 overall SSIs (sensitivity, 50.2%) and 73 of the 157 deep incisional and organ/space (DI/OS) SSIs (sensitivity, 46.5%). Also, 1 case reported to NHSN did not meet NHSN SSI criteria.
A review of records with claims codes suggestive of an SSI identified 201 of the 239 overall SSIs based on NHSN criteria (sensitivity, 84.1%) and 138 of the 157 DI/OS SSIs based on NHSN criteria (sensitivity, 87.9%). These data corresponded to a workload efficiency of 1 SSI detected for every 2.6 patients reviewed based upon the presence of a claims code suggestive of an SSI following colon surgery (PPV, 38.5%).
Table 3 compares differences in the NHSN surveillance criteria among SSIs identified versus SSIs missed by traditional hospital surveillance. Compared with cases identified by traditional surveillance methods, the SSI cases following colon surgery that were only identified by claims-based surveillance more often occurred during the index (surgical) admission (76% vs 58%; P<.01) and less often had a positive, aseptically obtained culture of fluid or tissue taken from the incision or organ space (55% vs 69%; P=.02). Documentation of an SSI by the surgeon or other attending physician was similar in those cases that had been identified versus missed by traditional surveillance (68% vs 72%; P=.57). The presence of purulence was also similar in these 2 groups (55% vs 54%; P=.90).
TABLE 3 Surveillance Criteria for Surgical-Site Infections (SSIs) Identified During Validation Project
NOTE. SSI, surgical-site infection; NHSN, National Healthcare Safety Network.
Overall, claims-based surveillance identified unreported SSIs in 31 of the 42 validation hospitals (74%), including 7 hospitals that had previously reported no SSIs. It should be noted, though, that claims-based surveillance is used to augment traditional surveillance and is not intended to be used as a replacement. In the case of colon surgery, 38 SSI cases were confirmed by the CPDH infection preventionists that had not been flagged for review by a billing code suggestive of SSI; 24 (63%) of these cases had been documented as SSIs by the surgeon or other attending physician.
Surveillance for SSI Following Abdominal Hysterectomy
The right column of Table 2 shows the performance of traditional hospital surveillance versus claims-based surveillance for identifying SSIs following abdominal hysterectomy. During the on-site validations, the CDPH infection preventionists reviewed full-text medical records for 133 patients who had undergone abdominal hysterectomy, including 22 patients identified by traditional surveillance only, 32 patients identified by both traditional surveillance and claims-based surveillance, and 81 patients identified by claims-based surveillance only. Overall, the CDPH infection preventionists identified 76 SSIs (30 superficial incisional SSIs, 12 deep incisional SSIs, and 34 organ/space SSIs). Of these 76 SSIs, 11 (14%) occurred during the same admission as the surgical procedure, and 65 (86%) were identified post discharge.
Traditional hospital surveillance performed prior to the CDPH validation visit had identified only 52 of the 76 overall SSIs (sensitivity, 68.4%) and 31 of the 46 DI/OS SSIs (sensitivity, 67.4%). In addition, 2 of the cases reported to the NHSN did not meet NHSN SSI criteria.
A review of records with claims codes suggestive of an SSI identified 56 of the 76 overall SSIs based on NHSN criteria (sensitivity, 73.7%) and 36 of the 46 DI/OS SSIs based on the NHSN criteria (sensitivity, 78.3%). These data corresponded to a workload efficiency of 1 SSI detected for every 2.0 patients reviewed based upon the presence of a claims code suggestive of an SSI following abdominal hysterectomy (PPV, 49.6%).
Table 3 compares differences in the NHSN surveillance criteria among SSIs identified versus missed by traditional hospital surveillance. Overall, there were no significant differences between the SSI cases following abdominal hysterectomy that were identified versus missed. Again, documentation of an SSI by the surgeon or other attending physician was similar for cases that had been identified versus missed by traditional surveillance (69% vs 67%; P=1.00), as was the presence of purulence (54% vs 50%; P=.81).
Overall, claims-based surveillance identified unreported SSIs in 12 of the 34 validation hospitals (35%), including 6 hospitals that had previously reported no SSIs. It is again important to highlight, though, that there were 20 SSI cases confirmed by the CDPH infection preventionists which had not been flagged for review by a billing code suggestive of an SSI, and 16 (80%) of these cases had been documented as an SSI by the surgeon or other attending physician.
Evaluating the Added Benefit of Antibiotic Surveillance
Only 19 hospitals submitted optional data on antibiotics in the 30-day postoperative surveillance window. A review of records of patients who had received ≥8 days of antibiotics following their colon surgery identified 40% (8 of 20) of the previously reported cases missed by claims-based surveillance in these hospitals, but the CDPH infection preventionists had to review an additional 5–6 patient records flagged only by prolonged antibiotic days for each SSI identified (PPV 18%). For abdominal hysterectomy, reviewing records of patients who had received ≥4 days of antibiotics following their surgery identified no additional cases identified by traditional surveillance but missed by claims-based surveillance.
ICD-9 to ICD-10 Crosswalk
Table 4 shows the ICD-9 to ICD-10 crosswalk for our recommended SSI validation codes. We separately analyzed the performance of claims-based surveillance in the colon validation using only ICD-9 diagnosis codes, given the complexity of mapping ICD-9 procedure codes to ICD-10 procedure codes.Reference Calderwood, Kleinman and Bratzler 21 , Reference Calderwood, Kleinman, Murphy, Platt and Huang 22 After dropping ICD-9 procedure codes 54.0 (incision and drainage of abdominal wall), 54.11 (exploratory laparotomy), 54.19 (drainage of intraperitoneal abscess or hematoma), 86.04 (skin and subcutaneous incision and drainage), 86.22 (excisional debridement of wound, infection, burn), and 86.28 (nonexcisional debridement of wound, infection, burn), the ICD-9 diagnosis codes identified 194 (97%) of the 201 SSIs identified by the full code set. These data changed the sensitivity from 84% to 81% (and from 88% to 85% for DI/OS SSIs). We also dropped ICD-9 diagnosis code 879.9 (open wound(s) (multiple) of unspecified site(s), complicated) because it was never used in our validation cohort and maps to an ICD-10 diagnosis code that is unrelated to SSI (S31020A, Laceration with foreign body of lower back and pelvis without penetration into retroperitoneum, initial encounter). A similar comparison was not done for abdominal hysterectomy because the claims-based code set for SSI detection only includes diagnosis codes.
TABLE 4 International Classification of Diseases, Ninth Revision (ICD-9) to ICD-10 Crosswalk for Recommended Surgical-Site Infection (SSI) Validation Codes
DISCUSSION
While CDC SSI surveillance definitions are standardized, we found that SSI case finding differs across hospitals and fails to identify one-third to one-half of all SSIs. This failure led to underreporting of SSI events in 74% (colon surgery) and 35% (abdominal hysterectomy) of the hospitals that participated in the 2013 California Department of Public Health Validation Project. Requiring that postoperative billing codes (ICD-9/ICD-10 diagnosis codes) be incorporated into routine surveillance will improve the sensitivity and efficiency of case finding and will improve the validity of publicly reported comparisons of hospital surgical performance. We demonstrate that this methodology can be effectively used for external validation of publicly reported data.
Validation of publicly reported SSI data is necessary to drive comparable case capture across hospitals. Without such validation, hospitals that perform more robust surveillance may be unfairly penalized by hospital performance measures linked to these data. At the same time, state and federal laws mandating reporting of healthcare-associated infections have led to increasing collaboration between hospitals and departments of health,Reference Stone, Pogorzelska-Maziarz and Reagan 23 along with greater attention from hospital leadership focused on preventing targeted HAIs.Reference Lee, Hartmann and Graham 24 Therefore, it is critical that we focus on the benefits of public reporting while understanding and addressing some of the limitations of the publicly reported data.
The 2013 California Department of Public Health Validation Project identified some key areas for hospitals to focus on in improving SSI surveillance. First, most of the SSIs following colon surgery occurred during the same hospitalization as the surgical procedure. They were often managed without reoperation, and they lacked positive microbiology in ~30% of cases. Because readmission, reoperation, and positive microbiology are key triggers for traditional surveillance, cases in which these factors are absent provide good examples of where claims-based surveillance can augment traditional methods. As discussed, antibiotic surveillance may further augment traditional surveillance, although our findings suggest that this methodology is much less efficient and the labor costs to review additional charts may not be worth the small improvement in case identification.
For SSI surveillance following abdominal hysterectomy, most SSIs were also managed without reoperation, and they lacked positive microbiology in ~30% of cases. Unlike colon surgery, however, 86% of the SSIs following abdominal hysterectomy occurred after the patient had been discharged following surgery, with 62% of these patients requiring readmission to the hospital. To improve case capture, hospitals can use claims-based surveillance to target case review of these hospital readmissions. In addition, while prior work suggested some benefit from antibiotic surveillance,Reference Yokoe, Khan and Olsen 14 we did not find similar benefit in this study in terms of identifying cases previously identified by infection control but missed by claims-based surveillance.
Our study does have some limitations. First, our sampling disproportionately included teaching hospitals and hospitals in Southern California, particularly in Los Angeles county. The hospitals that volunteered to have their SSI data validated included only 2 of 63 rural hospitals, 1 of 28 critical access hospitals, and 1 of 12 pediatric hospitals in the state of California. At the same time, though, 64% of the volunteer hospitals were nonteaching hospitals, suggesting that the results are generalizable to community hospitals in addition to academic medical centers. Second, we did not review charts other than those identified by claims suggestive of SSI or previously identified by traditional surveillance methods prior to the validation visit. It is possible that other methods such as reviewing all readmissions regardless of codes suggestive of an SSI or review of records with a prolonged surgical admission might further augment case detection, although it will be important to evaluate the workload efficiency (positive predictive value) of these alternate methods. Third, claims-based surveillance relies on hospital billing codes, which are typically not available until a patient is discharged from the hospital. Thus, this type of surveillance is often retrospective rather than real time. For this reason, many recommend continuing traditional surveillance methods while adding enhanced surveillance methods such as claims-based surveillance. Finally, it is possible that hospitals could learn to manipulate the system by changing billing practices. This problem was previously demonstrated when financial reimbursement programs targeted limited codes as indicators of specific HAIs.Reference Calderwood, Kleinman and Soumerai 25 , Reference Kawai, Calderwood and Jin 26 Our code sets purposely include a larger number of billing codes to account for variable use of specific codes across hospitals.
In conclusion, our findings strongly support the use of diagnosis claims codes to trigger case finding through chart review for NHSN SSI cases following colon surgery and abdominal hysterectomy. We also confirmed the utility of external validation by public health departments to ensure appropriate capture and reporting of SSIs in the NHSN system. The use of claims codes and validation of these publicly reported data can help individual hospitals improve their surveillance to respond to elevated SSI rates and can help ensure fairer comparisons across hospitals. CDPH has made available to all California hospitals a set of ICD flag codes for each of the 29 reportable surgical procedures and has recommended that they be used as part of routine SSI surveillance. 27
ACKNOWLEDGMENTS
Financial support: Drs Michael S. Calderwood and Susan S. Huang received grant support from the Agency for Healthcare Research and Quality (grant no. R18 HS021424).
Potential conflicts of interest: All authors report no conflicts of interest relevant to this article.
