The issue of assessing the disaster preparedness of a health care organization is a topic that is of great interest from the perspective of both providers and researchers. One basic problem highlighted in the literature, however, is the lack of a uniform definition of a disaster.Reference Perry and Qurantelli 1 Studying the response to a disaster may lead this process forward. This necessitates a method by which to evaluate an organization’s preparedness as expressed in general terms, rather than an organization’s specific preparedness for each of the many situations that may occur.
Decisions and actions are therefore needed to redistribute resources, making the management of a major incident one of critical processes, and thus an important part also of disaster preparedness. There are, however, limited ways of assessing the management part of disaster preparedness. One way is to evaluate the performance of management groups in simulation exercises. 2 If both structure and process could be evaluated in the same exercise, this could lead to a better estimation of the level of preparedness. The Hospital Incident Command System (HICS) exemplifies one way of assessing a hospital’s general disaster preparedness. The HICS model, previously proven to be useful and allowing comparison between different hospitals in different settings, contains a standardized way to measure actions for management group functions, which can be used to measure the structure and performance level of each function.Reference Born, Briggs and Ciraulo 3 - Reference Zane and Prestipino 5 The HICS addresses both structural and process indicators of a management group performance.
Another method, a more-process oriented method that has been put forward, is the Disaster Management Indicator model (DiMI). This model, which addresses processes and structures, has been used in several educational contexts and also for evaluation of real incidents.Reference Nilsson, Vikstrom and Ruter 6 , Reference Nilsson, Vikstrom and Jonson 7
DiMI only addresses whether a decision, based on a modeling process, was performed but does not assess the effect of each decision because this further step requires a qualitative approach also. An issue to be addressed is if the 2 methods are used together, do the results from these models correspond or do the results diverge? The aim of this study was thus to highlight 2 different tools for evaluating the in-hospital management of a disaster situation in simulation exercises.
Methods
Setting
Two tabletop exercises were conducted at 2 major hospitals in the Stockholm area on September 18 and 19, 2012.
Scenario
The simulation exercises were conducted with the aim of evaluating medical response to a mass casualty incident: an explosion in the center of Stockholm. At both hospitals, the Emergo Train model was used as the simulation tool. 8 The Emergo Train System (ETS) is an interactive simulation system developed in Sweden. It can be used for education, training, and simulations of emergencies and disasters. The ETS can be used to test and evaluate incident command systems, hospital preparedness, and surge capacity.
Performance Indicators
The HICS performance indicators come from worksheets describing definitive positions. 2 Each position has a job action sheet consisting of multiple actions to be fulfilled by the responder in charge. Achievement of performance of each position is evaluated on the basis of accuracy of relevant actions taken by the responder, which was scored as a percentage.Reference Djalali, Castren and Hosseinijenab 4
The total HICS performance score was calculated as an average of the performance scores of all incorporated positions, also as a percentage. The HICS also demonstrates how many and which of the recommended positions had been filled during the exercise.
The DiMI model consists of 2 groups of indicators: (1) hospital management and (2) staff skills. Each group includes 11 indicators that are scored 2, 1, or 0 as correct, partly correct, and incorrect/omitted, respectively.
Evaluation
The evaluation focused on hospital immediate response within the first few hours after the incident. Both exercises were observed and evaluated by 2 researchers, with HICS and DiMI model experience, observing actions, processes, and structures. All participants of the simulations (staff and managers) were asked to document their decisions and actions. These documents were, together with the documented observations by the researchers, also used in the evaluation process. Final scoring and analysis of the results was done through consensus of both researchers; the results of the models were then calculated and analyzed.
Another evaluation criterion was the percentage of activated positions at the hospital compared to the 22 positions suggested by HICS-2006 for the immediate phase after an incident consisting of an explosion. A position is equivalent to a functional area; thus, it was possible for a staff member to man more than one position.
Ethical consideration
The evaluation was position-specific as opposed to person- or participant-specific, which made approval from an institutional review board unnecessary. The Helsinki declaration was followed.
Results
Exercise A indicated that of 22 HICS-2006 positions suggested for the immediate phase after an explosion, only 7 (32%) were manned by the hospital emergency response system.
In exercise B, 9 of 22 HICS-2006 positions suggested for the immediate phase after an explosion were activated (41%). In both exercises, 2 positions not belonging to the immediate phase, mental health and documentation, were also activated. In both exercises, the number of persons in the management staff varied over time (10-15 persons) owing to persons working in shifts and replacing each other.
In both exercises (A and B), the activated positions were mainly from the command team and the operation section; security branch positions, however, were not activated (Table 1).
Table 1 Activated Positions and Performance of the Hospital Response System, on the Basis of the HICS Model in Exercises A and BFootnote a
a Abbreviation: HCIS, Hospital Incident Command System.
On the basis of the HICS indicators (Table 1), the average performance of the activated positions, concerning the hospital response team, was 70% in exercise A and 68% in exercise B. According to the DiMI model, the managerial performance (Table 2) of the hospital response system was 68% (15/22) in exercise A and 77% (17/22) in exercise B. The hospital performance, on the basis of staff skill indicators, was 63% (14/22) in exercise A and 86% (19/22) in exercise B.
Table 2 Performance Indicators, Staff Skills, and Management on the Basis of the DiMI Model, in Exercises A and BFootnote a
a Abbreviation: DiMI, Disaster Management Indicator model; ICU, intensive care unit. The indicators were scored as follows: correct, 2 points; partly correct, 1 point; incorrect, 0 points.
Discussion
Finding methods for evaluating disaster preparedness is a challenge. One way is to assess preparedness on the basis of an after-event report based on a template.Reference Debacker, Hubloue and Dhondt 9 Another way is to address structures by use of a holistic approach based on indicators on several organizational levels. 10 By using results obtained from performance during exercises, this study highlights a more limited way of addressing disaster preparedness.
The results from the present study indicate that there is no single method that solely provides a full picture of the level of preparedness based merely on simulation exercises. The HICS model addressed the structural parts more completely than did the DiMI and also demonstrated a higher percentage of activated positions in exercise A than in exercise B. HICS has previously been used to evaluate hospital standard operating procedure.Reference Djalali, Castren and Hosseinijenab 4 In that previous study it was suggested that HICS be used by hospitals in Sweden, as well as in other countries, owing to its (relative) comprehensiveness. However, a good structure is only one key factor; there is also a need for processes and actions to take place. In this respect, both systems indicate somewhat contradictory results. DiMI gave better scores for exercise B, and HICS was slightly better in exercise A. With respect to the fact that the 2 models did not address the same issues, however, caution is urged regarding these conclusions in as much as this was a pilot study, and no inferential statistics were performed.
It must, however, also be understood that there is no accepted standard of best performance with which to make comparisons. For this there is a need for evaluating performance also during situations other than simulations. Also, drawing conclusions from the somewhat diverse results cannot be done after 2 exercises only. The difference between HICS and DiMI with regard to processes is that in the DiMI model there is also a standard (a suggested benchmark) set to each process indicator. The scoring of the result as 0, 1, or 2 will thereby have a substantial impact on the total score. These standards, expressed as time limits, have been used in several contexts, but have yet to be validated. However, 2 models may be complementary and do indeed to a certain degree give the same information. The results of the structure part of DiMI (staff skills) point in a direction opposite to that of HICS, and this needs to be studied further. One explanation could be that the structure issues related to the 2 models have a different focus, and perhaps this should be interpreted as different aspects being evaluated. It is interesting to note that although positions were manned lower according to HICS in exercise A (9 of 24) than in exercise B (11 of 24), the results differed in percentage, being higher in exercise A.
The management score in exercise B was higher than in A, which could suggest that the manning 11 of 24 positions according to HICS (2 positions are not considered as immediate, thus 24), could be related to what was achieved. Two exercises, of course, are not enough on which to draw conclusions on disaster preparedness. Many other factors could have been involved.
The results from this pilot study in which 2 different tools for evaluation were used suggest that the 2 applied models for estimating disaster response as a part of disaster preparedness could complement each other, although more studies are needed to substantiate these findings. However, methods for assessing disaster preparedness must continue to be developed. Any shortcomings of preparedness must be identified beforehand, and if possible eliminated or at least mitigated, even if this means the investment of time and money. If not, the actual costs as a consequence of an incident may be higher, and it is also likely that the management of patients will be affected.
Conclusions
Results from 2 similar exercises, evaluated by use of the HICS and DiMI model, demonstrated performance on an acceptable level in relation to results from previous studies. However, most of the positions were missed according to the HICS method. More research on this area is needed to validate which of these methods best evaluates disaster preparedness based on simulation exercises or whether these methods are complementary and therefore should be used together.
