Introduction
The objective of disaster preparedness is to ensure that appropriate systems, procedures, and resources are in place to provide prompt, effective assistance to disaster victims, thus facilitating relief measures and rehabilitation of services. Disaster preparedness is an ongoing, multisector activity that should take into account the vulnerability of a specific country or region and its health system infrastructure and capabilities.Reference De Ville de Goyet, Monje and Poncelet1 Disaster preparedness efforts include the identification of possible health scenarios based on the probability of hazards and vulnerability of the population as a basis for creating a disaster plan. Exercises that simulate emergency response, involving the health and other sectors, have been suggested as useful tools to test disaster plans on a regular basis and measure preparedness efforts; the absence of actual testing is likely to negate even the best of abstract plans.Reference Donahue and Tuohy2-Reference Stoto and Nelson6
Unfortunately, the variability in the quality of the exercises conducted and the lack of standardized processes to measure performance have limited the value of exercises to document the return on governmental or institutional investments with respect to preparedness expenditures. In particular, practitioners’ use of written documents such as after action reports (AARs) to describe lessons learned during the conduction of an exercise or in response to a real event has often been unhelpful for systematic learning.Reference Donahue and Tuohy2, Reference Savoia, Agboola and Biddinger7 Part of this challenge may be attributed to the fact that there is no evidence on what makes an exercise an effective tool to assess public health emergency preparedness (PHEP), and how AARs can be developed and used to support PHEP improvement efforts. To partially fill this knowledge gap, two groups of PHEP experts were convened to respond to the following questions: (1) what makes an exercise an effective tool to assess PHEP; and (2) what makes an AAR an effective tool to guide PHEP improvement efforts?
Methods
The study was conducted in the US, and opinions from 61 public health officials and emergency responders were systematically gathered and analyzed by the use of two nominal group techniques (NGTs) in July 2009 and July 2011.8, Reference Delbecq, VandeVen and Gustafson9 Panelists were selected as a convenience sample of practitioners with experience in emergency preparedness exercises. To limit selection bias, different layers of government, as well as variety in the size of the jurisdiction served by their agency, were taken into consideration. The study included representation from federal agencies, state and local health departments (LHDs), large and small jurisdictions, urban and rural areas, and various US regions. The groups discussed the types of financial, organizational, networking, and system level barriers experienced by health departments while implementing the exercises and how exercises and AARs can be effectively developed and integrated in quality improvement efforts. Each NGT was guided by trained facilitators and organized so that participants could first develop a ranked list of issues and subsequently discuss and agree upon the list and their recommendations as a group.
Results
What Makes an Exercise an Effective Tool to Assess PHEP?
Panelists reached consensus on the characteristics that exercises should have when designed to assess PHEP, of which the most frequently named were the inclusion of: (1) practitioners playing key leadership roles in the “real world” and (2) representatives from agencies and disciplines across the range of jurisdictions that would respond to a specific public health threat. Panelists commented that when senior level players are absent, altered decision-making processes are observed, either with respect to the types of decisions or the choosing of the appropriate hierarchical level for the decision being made. Similarly, if specific agencies do not participate, inaccurate assumptions about the agencies’ response role and capabilities are made. Panelists highlighted the importance of having a plausible scenario and timeline in the conduction of the exercise to test actual capabilities and maintain participants’ engagement, and the need to design clear and measurable exercise objectives named at the outset of the planning process. Mistakes in adhering to this process may lead to the development of an inept exercise scenario and may compromise the agencies’ performance evaluation.
Furthermore, panelists argued that, mainly due to lack of resources, exercises are too often conducted as one-time events with objectives not linked to prior years’ efforts and prior-tested capabilities, thereby limiting the ability to document improvement over time. Moreover, panelists reported that expertise in exercise planning is a limited or unavailable resource in local health departments (LHDs). During the first NGT session, panelists reported that the key to a good evaluation process is the availability of a sufficient number of trained and competent external evaluators who are capable of identifying the root causes of the response failures observed. They noted that while exercises are most commonly recommended for evaluation, in practice, they are frequently conducted with multiple goals in mind, including planning, training, relationship-building, and evaluation. While these goals are interrelated, and there may be occasional opportunities to achieve them concomitantly, heterogeneity in scopes has implications for the exercise design, the required level of participation, and the approach to evaluation. Consensus achieved by the first NGT session is summarized in Figure 1.
Figure 1 Recommendations for Exercise and AARs Used for PHEP Evaluation Purposes Abbreviations: PHEP, public health emergency preparedness; AAR, after action report
What Makes an AAR an Effective Tool to Guide PHEP Improvement Efforts?
With the understanding that, when properly designed, executed and evaluated, exercises can be useful to improve local and national preparedness plans and systems, 30 PHEP experts joined a second NGT session to discuss how AARs should be structured, completed, and aggregated to contribute to quality improvement (QI) efforts. The panelists began with identifying barriers to the use of evaluation data derived from exercises in order to describe system performance in AARs. Frequently, exercise objectives and evaluation instruments are developed independently from one another. As a result, the data gathered during the exercise and subsequently used to write the AAR may be poorly suited to describing whether or not the stated objectives were met. Additionally, observations are often presented without providing root-cause analysis of specific response failures. Specifically, AARs rarely include a structured approach to identify the factors that resulted in the nature, the magnitude, the location, and the timing of a specific response failure.
Panelists commented that exercise planners should prospectively match the stated exercise objectives with the capabilities to be tested for each exercise, and then use both exercise objectives and capabilities to describe agencies’ performances in the AARs. As an example, if exercise planners have the exercise objective of evaluating the ability of a local hospital to communicate information about a possible new outbreak of a highly infectious disease to their health authority, those planners should prospectively design an exercise evaluation tool that measures data on the capabilities (domains) of biosurveillance, information sharing and information management. In the AAR, data collected on those capabilities would then be used to describe and critically evaluate the hospital's and health department's performance based on the exercise objectives. This mechanism would improve the process of identifying the causes of response gaps and name necessary improvement actions. Panelists also commented on the dual nature of AARs, frequently used for both accountability and QI purposes. When AARs are used for accountability, public health practitioners are typically eager to show their strengths. When AARs are used for QI purposes, practitioners are more interested in identifying pitfalls that may forecast failures in the response to future incidents so that those errors can be corrected. It is not ideal to have one document to serve both purposes, so several panelists suggested creating two different documents, taking into consideration the already limited time and effort available for writing AARs at the local level. Panelists reported that AARs could be used to better support repeated evaluation of similar objectives and capabilities and to document improvement over time if linked to Multi-Year Training and Exercise Plans (MYTEPs). Moreover, AARs could strengthen systematic learning if better organized and indexed by response capability. If the most common capability gaps which have routinely proved to cause response failures could be identified within regionally and nationally aggregated pools of AARs, such information could be used by practitioners when drafting their own local planning priorities and MYTEPs. Finally, panelists discussed the importance of developing strategies such as providing financial incentives, enhancing regulatory guidance, and/or publicizing best AAR practices to encourage the production of higher quality AARs. The panelists also acknowledged, though, that in the current fiscal environment, the incentives likely may not be substantial enough to motivate change. The consensus achieved by this second NGT session is summarized in Figure 1.
Discussion
The ultimate goal of conducting PHEP exercises is to improve a community's ability to prepare for and respond to emergencies. Producing “good exercises and good AARs” with the characteristics named by the panelists remains a challenge for most public health agencies due to fiscal, logistical, political, and other current issues within public health departments. More specifically to the US where the study was conducted, some of the challenges are also due to the fact that federal and other agencies require exercises and AARs for accountability purposes, overshadowing the role of these tools for improvement efforts. Fear of the consequences of naming errors in the response can influence exercise designers, participants, and AARs developers as they measure and document agencies’ and systems’ performance during exercises. This, of course, limits their subsequent ability to develop improvement plans and to provide an accurate picture of public health systems’ response capabilities. While the recommendations provided in this report are valid for any institution around the world interested in designing effective exercises and AARs for evaluation purposes, there remains the challenge of national and local authorities to identify appropriate incentives for local practitioners to produce high quality exercises and AARs that can support the PHEP QI cycle and provide a more accurate assessment of PHEP system capabilities.
Conclusions
The characteristics of the exercise audience, scenario, and scope are important attributes to the effectiveness of an exercise conducted for PHEP evaluation purposes. The evaluation instruments used to gather observations need an appropriate matching between exercise objectives and the response capabilities tested during the exercise, to build the base for the production of a good AAR. Improvements in the design and creation of exercises and AARs could facilitate better reporting and measurement of preparedness outcomes.
Acknowledgments
The content as well as the views and discussions expressed in this report are solely those of the authors and do not necessarily represent the views of any partner organizations, the CDC or the US Department of Health and Human Services nor does mention of trade names, commercial practices, or organizations imply endorsement by the US Government. The authors thank Dr. Michael A. Stoto for facilitating two of the nominal groups and Lindsay Tallon for her contribution to the analysis of the data.
