Introduction
Improving surgical safety has become a high priority in recent years, in order to reduce adverse events to patients. The World Health Organization (WHO) has identified surgery as the most complex intervention in healthcare. Its delivery leads to almost half of all avoidable adverse events that result in death or disability, and these may be reduced through the mitigation of peri-operative risks by confirming the patient's identity and allergy status and the site of surgery.1
The general public expect risks to be minimised, particularly in safety-sensitive industries such as healthcare and aviation. Checklists are one of the tools used by pilots to improve safety; their use in aviation predates the Second World War and has been shown to reduce serious incidents.Reference Degani and Wiener2
In 2008, the WHO ‘World Alliance for Patient Safety’ developed and published data on the efficacy of a ‘Surgical Safety Checklist’.Reference Haynes, Weiser, Berry, Lipsitz, Breizat and Dellinger3 The Checklist formed part of the ‘Safe Surgery Saves Lives’ campaign that aimed to highlight the importance of safe practice in the operating theatre, both in surgical and anaesthesia protocols. In 2009, Haynes et al. published trial data collected from eight diverse hospitals, from developed and developing countries around the globe, and demonstrated that the introduction of the WHO Surgical Safety Checklist reduced post-operative mortality of surgical patients from 1.5 to 0.8 per cent (p = 0.003), and post-operative morbidity from 11.0 to 7.0 per cent (p = 0.001).Reference Haynes, Weiser, Berry, Lipsitz, Breizat and Dellinger3 The report by Haynes et al. provided strong evidence for the efficacy of this simple intervention in terms of reducing the complications associated with surgery, regardless of geographical and socioeconomic factors. Most UK surgical units have adopted the WHO Checklist following guidance from the National Patient Safety Agency.4
The implementation manual, published in parallel with the WHO Surgical Safety Checklist, suggests that the Checklist should be adapted to fit in with local practice.5 The Wexham Park Hospital (Slough, UK) Checklist can be seen in Figure 1.
Fig. 1 The Wexham Park Hospital World Health Organization (WHO) ‘Surgical Safety Checklist’.
In this study, we aimed to prospectively assess compliance with the WHO Surgical Safety Checklist in our otolaryngology department operating theatres (at Heatherwood and Wexham Park Hospitals, UK).
Materials and methods
To establish compliance with the local WHO Surgical Safety Checklist, our otolaryngology operating theatre staff were assessed performing the ‘Sign in’, ‘Time out’ and ‘Sign out’ phases of the Checklist. Otolaryngology trainee doctors recorded data whilst observing operations over a six-week period. The assessments were conducted at random and were dependent on the availability of otolaryngology trainees in the operating theatre.
To minimise bias, operating theatre teams were neither informed, nor aware, that they were being audited. We designed and used a special proforma to assess compliance with the Checklist (Figure 2). Audit one provided a baseline of performance. Following an educational intervention, the audit was repeated 12 months later (audit two), using the same proforma and method described above.
Fig. 2 Proforma designed to assess compliance with the World Health Organization (WHO) ‘Surgical Safety Checklist’.
We compared compliance between audits one and two across 11 separate sections of the Checklist. To determine if the results were statistically significant, we used Fisher's exact test (for within-group differences) and the two-tailed, unpaired Student's t-test (for overall differences between groups). The Statistical Package for the Social Sciences version 15 software program (SPSS, Chicago, Illinois, USA) was utilised. Differences were regarded as statistically significant if the p value was less than 0.05. Where relevant, the results were expressed as mean ± standard error of the mean, with 95 per cent confidence intervals.
Ethical considerations
This study was registered with the Heatherwood and Wexham Park Hospitals National Health Service (NHS) Foundation Trust audit department, and no formal ethics committee approval was required. Patients undergoing surgery at Heatherwood and Wexham Park Hospitals NHS Foundation Trust were approached to participate in the study, and their full consent was obtained prior to enrolment. The study did not receive any commercial sponsorship.
Results
In the first part of our study (audit one), we assessed 36 non-consecutive elective and emergency otolaryngology procedures using the proforma shown in Figure 2, between 6 January and 1 March 2010. The mean patient age was 34.1 ± 4.3 years. Documentation of the planned procedure at Sign in was not completed in 16 (44 per cent) cases. There were interruptions to Time out in 14 (39 per cent) cases, and the procedure was inappropriately started before Time out was completed in 12 (33 per cent) cases. The mean time for completion of Time out was 60 seconds. The Sign out was read out in only 2 (6 per cent) cases, it was completed in the operating theatre in only 6 (17 per cent) cases, and it was not fully documented in 15 (42 per cent) cases.
The second part of this study (audit two) was carried out following a hospital awareness campaign and staff education intervention (see Discussion for further details). For audit two, a further 36 non-consecutive otolaryngology procedures were assessed, between 11 April and 11 May 2011. The patient population for audit two was not significantly different to that of audit one (with a mean age of 31.9 ± 10.4 years; p = 0.85, unpaired t-test), and underwent a similar range of surgical procedures. The compliance results (Figure 3) showed that the pre-operative documentation relating to patient details and the planned procedure was completed in 36 out of 36 cases. Time out was better performed, with a reduction in the number of procedures commencing before Time out was complete (5 cases) and in the number of interruptions to this phase (only 2 cases). The average Time out duration had increased to 96 seconds, but this was not statistically significant (p = 0.92, unpaired t-test). Sign out was greatly improved: it was read out, completed appropriately in theatre and fully documented in 34 of 36 cases.
Fig. 3 Results of the audit cycle, demonstrating a significant improvement across several key areas of the World Health Organization ‘Surgical Safety Checklist’. *p < 0.001, Fisher's exact test for within-group differences (i.e. statistical comparison of audit one and two results within each group or section of the form). ns = not significant
Calculations of overall compliance, determined by a composite average of all 11 Checklist sections, revealed an average compliance of 63.7 ± 8.9 per cent in audit one, and a significantly improved compliance of 90.4 ± 2.7 per cent in audit two (p < 0.01; two-tailed, unpaired Student's t-test; 95 per cent confidence interval, 7.3 to 46 per cent improvement in compliance).
Discussion
There was no significant variation in terms of the case mix and patient demographics for the procedures observed and analysed, in either part of the study, which spanned a total of 17 months and 72 patients. The case mix and patient demographics were considered normal for a UK district general hospital otolaryngology department's operative population.
Our results revealed poor compliance with the local WHO Surgical Safety Checklist in audit one. Of specific concern, and a factor which lead to the premature termination of this part of the study, was the frequent failure (in 16 out of the 36 cases observed) to properly document the planned procedure and the site of surgery pre-operatively. Time out was poorly observed in 19 of the 36 cases, with interruptions in 14 cases, and commencement of the operative procedure occurred before Time out was complete in 12 cases. Sign out was even more poorly performed. This seemed to be viewed by operating theatre staff as a superfluous phase and was read out in only 2 cases. Six Sign outs were completed in the operating theatre, and only 21 of the 36 cases observed were fully documented.
The challenges of effectively implementing the WHO Surgical Safety Checklist have been highlighted previously by other UK surgeons.Reference Vats, Vincent, Nagpal, Davies, Darzi and Moorthy6 Vats et al. analysed data from St Mary's Hospital (Imperial College Healthcare NHS Trust, London) over a six-month period and found that the local WHO Checklist was correctly completed in only 424 of the 729 operations assessed.Reference Vats, Vincent, Nagpal, Davies, Darzi and Moorthy6 Thus, there was only 58 per cent compliance in the original UK ‘WHO Safe Surgery Saves Lives Study Group’ trial. It was this trial that provided the evidence for the Surgical Safety Checklist efficacy.Reference Haynes, Weiser, Berry, Lipsitz, Breizat and Dellinger3 Vats et al. pointed out that Sign out was ‘rarely done’ when Checklists were incomplete, and went on to identify other characteristics of a poorly utilised Checklist, including: the procedure being hurried, dismissive replies, and the absence of key staff during the Time out phase.Reference Vats, Vincent, Nagpal, Davies, Darzi and Moorthy6 These barriers to Checklist utilisation were also observed by our group in the first part of the study (audit one).
The second part of our study (audit two) was performed 12 months after the first, following a major hospital and operating theatre staff awareness and educational intervention. The results showed a significant improvement in overall compliance with the local WHO Surgical Safety Checklist (63.7 vs 90.4 per cent, p < 0.01). See Figure 3 for a comparison between audits one and two. Of note, the pre-operative documentation relating to patient identification and planned procedure details was completed in all 36 cases observed, an improvement to 100 per cent compliance. (Only 55.6 per cent of cases in audit one had complete documentation of the planned procedure, leading to the termination of the first part of our study.) Time out was better performed compared with audit one, with fewer interruptions (2 vs 14) and fewer procedures commencing before its completion (5 vs 12), and this phase lasted an average of 36 seconds longer in audit two (although this was not statistically significant; p = 0.92, unpaired t-test). However, we identified Time out as an area that still needed improvement. The most improved criterion in audit two (compared with audit one) was compliance with Sign out, which was read out in 34 cases (vs 2 cases in audit one), fully documented in 34 (vs 21) cases and completed in theatre in 34 (vs 6) of the 36 cases.
The results of our second audit are encouraging. They show a significant improvement in compliance with our Surgical Safety Checklist following a local staff educational intervention. However, we did not directly measure surgical outcomes such as complication rate, discharge delays or other adverse events related to hospital admission. We therefore cannot determine whether the improved peri-operative compliance with the Checklist led to a corresponding reduction in mortality or morbidity. Notably, Vats et al. reported that, despite observing an improvement in safety processes such as timely antibiotic prophylaxis administration, which rose from 57 to 77 per cent following the introduction of the Checklist, they did not find an improvement in mortality or morbidity.Reference Vats, Vincent, Nagpal, Davies, Darzi and Moorthy6 They attributed this to an already low complication rate at their hospital, which may raise questions over the universal benefits of the Checklist. It is therefore interesting to consider the expected benefits of the Surgical Safety Checklist in varied surgical settings, ranging from elective out-patient cases performed under local anaesthetic in a community setting, to emergency in-patient cases in a tertiary hospital trauma theatre.
The Safe Surgery Saves Lives Study Group has published a follow-up study to demonstrate the efficacy of their 19-point safety Checklist in urgent surgical procedures.Reference Weiser, Haynes, Dziekan, Berry, Lipsitz and Gawande7 Cardiothoracic surgery was omitted from the original WHO study, and its specific demands have necessitated adaptation of the WHO Checklist.Reference Clark and Hamilton8 It is encouraging that different UK centres and specialties are beginning to publish their experiences with these Checklists.Reference Sewell, Adebibe, Jayakumar, Jowett, Kong and Vemulapalli9–Reference Kearns, Uppal, Bonner, Robertson, Daniel and McGrady11 Different hospitals and surgical specialities may have different Surgical Safety Checklist requirements, which is the reason the WHO recommends adapting their Checklist template to meet local needs. Although the authors of the WHO Surgical Safety Checklist have recently published a report describing how they designed their original 19-point Surgical Safety Checklist,Reference Weiser, Haynes, Lashoher, Dziekan, Boorman and Berry12 and have previously provided useful guidance on how others can adopt it, they have not published guidance on how to effectively adapt the Checklist to local requirements.
We recommend better staff training and the identification of local senior ‘Checklist champions’ to remedy two of the important underlying barriers to Checklist compliance: the lack of awareness of the Checklist's efficacy, and a generally poor safety culture. Poor compliance is probably multifactorial, but our group demonstrated improvement in compliance after implementing the following changes. Firstly, we initiated a new local awareness campaign and retrained operating theatre and relevant ward staff on the Surgical Safety Checklist procedure. Secondly, we designated senior clinical Checklist champions to implement the procedure, educate other staff and act as agents of cultural change. Thirdly, we mandated the completion of documentation of the planned operation before Sign in, preferably on the ward as part of the patient's peri-operative care pack, before any patient was permitted to go to theatre by the nursing staff. Fourthly, Sign in and Time out were made the responsibility of the anaesthetist, who, upon satisfactory completion, was required to sign the relevant part on the Checklist. Fifthly, we ensured that all operating theatre staff strictly observed Time out with full attention while the anaesthetist read out and documented this phase. Finally, we ensured that Sign out was read out in the operating theatre at the end of the procedure by the lead or designated surgeon, who then documented and signed the relevant section on the Checklist; again, other theatre staff strictly observed this.
On the basis of our results, we also recommend continued auditing to assess compliance. Although compliance could and should be expected to improve, measurement of mortality and morbidity outcomes should also be attempted in order to assess any correlation. Interestingly, a study involving 965 patients from the Stanmore Royal National Orthopaedic Hospital again found no improvement in early complications (within 30 days after the operation) or mortality when comparing the initial low (7.9 per cent) Checklist compliance to their post-educational intervention high (96.9 per cent) Checklist compliance.Reference Sewell, Adebibe, Jayakumar, Jowett, Kong and Vemulapalli9 This study, like ours, shows that a dramatic improvement in Checklist compliance can be achieved with a local educational intervention, although the lack of a demonstrable associated improvement in morbidity and mortality rates may subsequently limit long-term compliance.
• The World Health Organization (WHO) ‘Surgical Safety Checklist’ was adopted by the UK following National Patient Safety Agency guidance
• We assessed compliance with the local WHO Checklist; the results revealed poor compliance
• This prompted a hospital awareness campaign and educational intervention
• A subsequent audit showed improved compliance, with alterations to local guidelines
The lack of improvement in terms of patient outcomes may be due to the fact that small improvements go undetected, and a greater number of patients need to be assessed to reveal these benefits (i.e. the study may have limited power). Furthermore, some units and specialties like otolaryngology may have a low baseline of patient harm and may not be able to easily demonstrate the efficacy of their local WHO Checklists.Reference Karamchandani and McGarry13 However, a recent survey-based study from a Finnish group found that implementation of the WHO Surgical Safety Checklist did improve communication between team members in otolaryngology operating theatres, as well as improving the discussion of important patient safety factors (e.g. allergy status) and the documentation of post-operative plans.Reference Helmiö, Blomgren, Takala, Pauniaho, Takala and Ikonen14 Benefits such as improved multidisciplinary team communication about patient safety and care are not easy to measure and may have undetectable beneficial effects on a unit's service provision. The Safe Surgery Saves Lives Study Group recently published a report in an attempt to demonstrate that better clinician attitudes towards patient safety correlated with a reduction in post-operative complications.Reference Haynes, Weiser, Berry, Lipsitz, Breizat and Dellinger15 Our group feels that further data using standardised outcome measures are needed from different surgical centres and specialties, to enable comparison and proper evaluation of the effects of the WHO Surgical Safety Checklist.
Conclusion
Our findings highlight the importance of assessing local WHO Surgical Safety Checklist compliance, and demonstrate the effectiveness of an educational intervention in improving compliance. We identified remedies to two important underlying barriers to compliance: the lack of awareness of the efficacy of a properly utilised WHO Checklist, and a generally poor safety culture amongst hospital staff. We advocate the continual commitment of senior clinicians and management to improve staff awareness and foster local ‘safety champions’. We also call for more data to be published by different hospitals and specialities on the efficacy of their local WHO Surgical Safety Checklists, both in terms of compliance, and potential effects on surgical outcomes and patient satisfaction.
Acknowledgements
We gratefully acknowledge all the patients who participated in this study. We also thank all the staff of the ENT department of Wexham Park Hospital, for making this study possible.
