The United States has made significant progress in public health and medical preparedness since 2001, but shortcomings continue in the areas of health care system preparedness for critical events and mass casualties. A large-scale weapon of mass destruction, a naturally occurring pandemic, or a devastating geological event could immediately overwhelm our medical infrastructure. Experts have recognized the need for improved hospital preparedness that addresses the full range of potential disasters,1 and the president of the United States has deemed it critical that a strategic vision be developed and implemented to enable a sufficient degree of public health and medical preparedness to address a range of possible disasters.2 Despite marked efforts to improve disaster planning, there remains no standardized, validated instrument to assess hospital emergency preparedness for all hazards.Reference Hogan and Burstein3, Reference Kaji, Langford and Lewis4
In 2004, following Homeland Security Presidential Directive 5,5 the US Department of Homeland Security developed the National Incident Management System (NIMS).6 NIMS provides a systematic, proactive approach to guide departments and agencies at all levels of government and the private sector to work together to prepare and respond to disasters. In addition, NIMS also provides a core set of emergency management principles and outlines organizational processes to enable effective, efficient, and collaborative incident management for all hazards. Directive 5 required hospitals and health care systems that receive any category of federal preparedness response funding to become fully trained in NIMS.
As hospitals become NIMS compliant, it remains unclear whether the current method of evaluating hospital preparedness addresses the NIMS components. The purpose of this article is to review instruments that have been developed to assess hospital emergency preparedness and evaluate how comprehensively these instruments measure NIMS elements such as preparedness, communication, information management, command, and management. The results of this study should further the development of a universal self-assessment tool that hospitals can use to guide their emergency preparedness efforts.
METHODS
Identification of Studies
A Medline search was performed for peer-reviewed articles published from January 1988 to August 2008 using the terms disaster, preparedness, bioterrorism, performance measures, quality indicators, assessment, validation, effectiveness, evidence based, measurement, science-based, improvement, criterion validity, strategies, and instruments. Additional instruments were found by examining references of studies initially identified. Nationally known experts in hospital disaster preparedness were also canvassed to identify evaluation instruments that may not be found or otherwise be cited in the peer-reviewed literature.
Eligibility of Studies
To be eligible for inclusion the instrument had to meet the following criteria: focus on hospital preparedness, be all-hazards focused rather than hazard specific, and encompass multiple dimensions of emergency preparedness. Instruments that addressed only the preparedness of state or local agencies such as public health departments or that addressed only 1 aspect of preparedness were excluded. Instruments were also excluded that assessed preparedness for a single hazard such as bioterrorism or chemical emergencies.
Analysis of Instruments
First, we reviewed the NIMS manual and identified the dimensions and important elements of NIMS from the manual. Detailed descriptions of each element are shown in Table 1. Second, each hospital preparedness instrument was evaluated by comparing the items contained in the tool with the NIMS elements. Two of the authors independently reviewed each instrument and rated how well the instrument addressed each NIMS element. Each of the above authors has either firsthand experience in hospital emergency preparedness or education and training in NIMS. Instruments developed before the development of NIMS but that addressed incident command systems (ICSs) and their components were given credit for evaluating command and management.
TABLE 1 NIMS Principles for Hospital Emergency Preparedness Instrument Comparison
For each NIMS element, the evaluator judged whether the instrument addressed the NIMS element at all (score of 0 if not), partially measured the element (score of 1), or completely measured it (score of 2). The 2 independent ratings were compared and if there was disagreement, it was discussed among the evaluators until consensus was reached. The scores for each element were totaled within each NIMS dimension and across dimensions to determine which instruments most comprehensively addressed NIMS principles. In addition to comparison with NIMS elements, each instrument was also evaluated for the mode of assessment (self-assessment, agency or organization assessment, or private) and the type of developer (government agency, other organization, or private author).
RESULTS
Fifteen instruments were identified that measured hospital emergency preparedness. Four instruments were excluded from the study because they addressed only 1 aspect of preparedness, such as biological or chemical emergencies.7–10 For example, the Agency for Healthcare Research and Quality (AHRQ) and the Government Accountability Office both developed instruments to evaluate bioterrorism preparedness, but these were excluded for evaluating only a single hazard.7, 8 Two other instruments were excluded because they did not address hospital or facility emergency preparedness.Reference Braun, Wineman and Finn11, 12 For example, the Centers for Disease Control and Prevention (CDC) instrument is designed for public health departments only and was excluded.12
After exclusions, 9 instruments were evaluated for content, method of administration, type of developer, and adherence to NIMS principles, as discussed below. The purpose of all of the included instruments was to improve hospital preparedness in all departments and across a variety of hazards. The majority of the instruments were developed by government agencies.13–18 These include the federal agencies Centers for Medicare and Medicaid Services (CMS),19 CDC,Reference Niska and Burt20 AHRQ,13 and Veterans Health Administration (VHA),16 and state agencies Texas Department of Health (TDH)14 and the South Carolina Department of Health (SCDH).15 One instrument was developed by the Association for Professionals in Infection Control and Epidemiology (APIC).17 Although the audience for this instrument is infectious control personnel, it uses an all-hazards approach and evaluates preparedness among a variety of hospital departments. The main method of development among all of the instruments was the utilization of expert opinion.
Four of the instruments evaluated were developed for the purpose of self-assessment by an individual hospital. 16,17,19 The person listed to perform the self-assessment was the hospital disaster manager or planner, with the exception of the APIC instrument,17 which was targeted toward infectious disease personnel for administration. One instrument, developed by the AHRQ in 2007, was for the purpose of assessment by states, localities, or hospitals.13 The Joint Commission (TJC) standards were the only instrument designed for the regular, repeated evaluation of a hospital’s emergency preparedness by an outside nongovernment organization.18 The instrument developed by Kaji et alReference Kaji, Langford and Lewis4 combined self-assessment followed by evaluation by an outside academic-affiliated team.
Comparison With NIMS Principles
Table 2 compares each instrument measured against the NIMS elements of preparedness, communication, information management, resource management, command, and management. Scores are given for each instrument’s total compliance with NIMS and the subtotal scores for the 4 major elements. The lowest scoring instruments (scores below 20) when compared against NIMS elements included the CDC Bioterrorism and Mass Casualty Survey,Reference Niska and Burt20 which was limited by too few questions on communication and information management as compared with the NIMS document. The Kaji instrumentReference Kaji, Langford and Lewis4 appears to score low due to a regional focus or jurisdiction-specific questions for preparedness that are less applicable to hospitals in other parts of the country.
TABLE 2 Comparison of Hospital Emergency Preparedness Instruments With NIMS Principles
Higher scoring hospital preparedness evaluation instruments include the APIC-developed instrument,17 the recently developed Emergency Management Chapter by TJC,18 and the VHA instrument.16 The remainder of the instruments with scores between 20 and 40 included the AHRQ Questionnaire for Healthcare Facilities,13 the TDH instrument,14 the SCDH instrument,15 and the CMS instrument.19
The 3 instruments created before the NIMS included the APIC instrument,17 which scored higher than most, and the TDH and SCDH instruments,14, 15 which scored in the 20s. Although the APIC instrument17 was developed before the NIMS, it included assessment areas in ICS that led to the higher score. Several topic areas were covered in the hospital preparedness instruments that were not addressed in the NIMS document. Hazard vulnerability analysis (HVA),14,16,18,19 mass fatality management, 4,13,16 evacuation planning,Reference Kaji, Langford and Lewis4,13–19 and mental health planning for staff 13,14,18 were addressed in several of the instruments evaluated and not specifically mentioned in the NIMS principles and elements. Other components of preparedness included in instruments such as the CMS checklist19 but absent from NIMS included the periodic review of the disaster plan, a mechanism for tracking patients, and the assessment of the psychological impact of the disaster on patients.
DISCUSSION
Although the need for improvement of hospital preparedness for critical events and public health emergencies has been recognized,1 the analysis above demonstrates that no comprehensive instrument or gold standard exists by which to measure a hospital’s compliance with the NIMS components. Given the lack of content uniformity among the instruments described above, no standard likely exists to measure general hospital preparedness. A comprehensive hospital preparedness instrument would be useful to improve the quality and reliability of preparedness across the nation. This study reflects the first generation of analysis of hospital preparedness evaluation using the NIMS components as the first guideline. The use of NIMS as a guideline for hospital preparedness may itself be questioned, and despite the development of NIMS and the importance placed upon it by governmental agencies, the funding stream to support its implementation has been weak. The requirement of NIMS compliance for federal funding also further elevates the need for an instrument that accurately measures the NIMS components in addition to general preparedness.
The hospital preparedness evaluation instruments evaluated above present a mixed picture of adherence to the principles and components of NIMS. In general, the principles of command, management, communications, and information management were the most challenging or least fulfilled by the majority of instruments. This may be the result of the marked level of detail given to these components in the NIMS document. In addition, the areas not addressed in NIMS but covered by many of the instruments present a challenge to the use of NIMS as the standard by which hospital preparedness should be compared. The goal of broad-based applicability of NIMS to health care and non–health care institutions may also hinder it in being considered a gold standard for comparison.
The use of NIMS as a reference standard by which to measure hospital preparedness may be hindered by the lack of hospital-specific content. Preparedness areas not addressed by NIMS but covered in the instruments evaluated included HVA, mass fatality management, evacuation planning, and mental health planning for staff. Most disaster preparedness experts agree that an HVA approach is both sound and proven in practice to be necessary.Reference Hogan and Burstein3, 18 Although NIMS does focus on preparedness across the full spectrum of potential incidents and hazard scenarios, the presence or process of a HVA is not specifically listed. The most frequently addressed topic that is not specifically listed as a NIMS element is the presence of an evacuation plan. Evacuation planning is mentioned in NIMS as a possible role for the operations section under command and management, but it is not listed as an element or principle of preparedness.
Instruments that score higher in NIMS compliance in our review 16–18 have several factors in common. Each instrument was from organizations or agencies (TJC, APIC, and VHA) with many years of experience in emergency planning, and each instrument has undergone multiple versions and revisions. These high-scoring instruments addressed the components of command and management thoroughly, including the operations, planning, and logistics functions of ICS. Preparedness in the areas of communications and information management was also addressed more completely in the high-scoring instruments. The TJC instrument18 is for use by outside reviewers; however, it is accessible by emergency planners as a preparedness guide. The APIC instrument17 is targeted primarily to infectious disease personnel and epidemiologists for use in their home institutions, but it covers many aspects of preparedness in entire hospitals. The VHA instrument16 was developed for self-assessment by hospital emergency planners to provide feedback before the larger assessment performed by the VHA itself.
The supporting literature describing the majority of these instruments does not suggest that NIMS was a primary factor influencing the content during development. The tools described here were not developed for the sole purpose of achieving NIMS, but rather to improve hospital disaster preparedness as a whole. The presence of multiple preparedness components (eg, mass fatality management, evacuation planning) not present in NIMS in a majority of the instruments again suggests that it is not a reference standard by which to measure the quality of preparedness. It should not be interpreted that these instruments do not measure hospital preparedness well if they do not address all of the components of the NIMS documents. A complete assessment of hospital preparedness will likely include multiple components not covered in NIMS.
Other key areas of preparedness have been identified by recent national strategic planning documents that could prove useful for developing a standard for hospital preparedness. These 5 key areas of preparedness can be applied generally to public health and medical preparedness and include preparedness for all potential catastrophic health events; coordination across levels of government, jurisdictions, and disciplines; a regional approach to health preparedness; engagement of the private sector, academia, and other nongovernment entities in preparedness and response efforts; and understanding the important roles of individuals, families, and communities. 21–23
Future research regarding the evaluation of hospital preparedness should ensure that regardless of the elements or areas addressed, the tool should meet standards for instrument review. Examples of these standards include reliability, validity, responsiveness, interpretability, and consideration of respondent and administrative burden.24 Disaster preparedness experts should develop tools that meet these criteria by incorporating such elements as evidence-based design and repetitive evaluation of instruments after implementation. To our knowledge, the hospital emergency preparedness instruments developed thus far have yet to be tested for their reliability, validity, or responsiveness to change.
The use of NIMS as a standard by which to compare hospital emergency preparedness evaluation instruments was hindered by the lack of its focus on health care institutions. NIMS compliance itself has not been shown to improve hospital disaster preparedness. In addition, although all of the members of our review group have training in NIMS, not every member has hands-on experience in hospital disaster planning. The members of the group and their ratings were not tested for intra- and interrater reliability, and the team only used 1 round of ratings among 3 members.
Conclusions
The instruments available and in use today for the evaluation of hospital preparedness have variable adherence to the principles and elements set forth in NIMS. This is especially evident in the areas of command and management and communications and information management. The use of NIMS as a standard remains questionable because of its lack of focus on the health care environment and incomplete list of pertinent elements. The development of new tools and standards for hospital preparedness should be encouraged and vetted in a multidisciplinary evidence-based process. Policymakers and emergency management stakeholders should encourage the continual reassessment of preparedness evaluation instruments based on not only an evidence-based process but also feedback from real-world events and best practice evaluations. The list of elements to consider is long, and research should be focused on providing evidence for their individual value and determining which elements have the highest utility in a resource-constrained environment.
Authors' Disclosures
The authors report no conflicts of interest.
