A mass casualty incident typically results in multiple casualties that often have complex injuries and place unique demands on health care systems.Reference Peleg and Kellerman 1 , Reference Timbie, Ringel and Fox 2 Managing mass casualty incidents is a challenge due to the uncertain and dynamic nature of these events. Adding to this complexity, decision-makers must act in the face of uncertainty and make rapid decisions in real time.Reference Peleg, Michaelson and Shapira 3 , Reference Mackaway-Jones 4
Research studies related to decision-making under uncertainty have been explored by multiple industries, primarily through theories of organization and cognition.Reference Hodgkinson and Starbuck 5 KleinReference Klein 6 presents the naturalistic decision-making approach, which emphasizes the role of experience in complex decision-making in real-world settings. The naturalistic decision-making approach notes that experience enables people to rapidly categorize situations to make effective decisions. Klein found that within a military setting the naturalistic decision-making approach improves performance and supports the development of decision-support technological aids.Reference Klein 6 In addition, the naturalistic decision-making approach has improved training that is focused on decision requirements.Reference Klein 6
Additional research studies that focus on decision-making in disasters present key findings from descriptive studies or apply decision-support frameworks to actual or simulated events.Reference Djalali, Castren and Hosseinijenab 7 - Reference McLennan, Holgate and Omodei 11 Glick and BarbaraReference Glick and Barbara 8 conducted structured interview surveys with US Federal Coordinating Officers to define decision-making processes and influential decision-making factors during the initial response period in a presidentially declared Stafford Act disaster. This study’s results provided a decision-making process and presented influential factors such as previous knowledge and experience; degree at which the disaster situation is atypical (eg, hazard, severity); quality, amount, and speed of data; ability to integrate data into a mental framework; and urgency. This study purported to support future research on identifying influencing processes and factors to assist with decision-making in high-consequence disasters.Reference Glick and Barbara 8
Furthermore, advancements in operations research attempt to address resource allocation, triage, and prioritization of the distribution of patients during a mass casualty incident primarily through modeling.Reference Amram, Schuurman and Hameed 12 - Reference Zuerlein 23 Obtaining expert agreement on whether the model inputs were influential to decision-making in mass casualty incidents was not explored. As noted by Klein,Reference Klein 6 and applying the naturalistic decision-making approach, it is important to explore knowledge identified through experience by experts.
No research studies were found that sought agreement by peer reviewers on factors that may influence decision-making related to the distribution of patients during a mass casualty incident. The aim of this formative study was to identify and seek agreement on factors that may influence decision-making related to the distribution of patients during a mass casualty incident.
METHODS
A two-round modified Delphi study was conducted by using a purposive sample of disaster management peer reviewers. A modified Delphi study is a research technique that aims to seek agreement on specific factors among a group of experts and can be conducted through a consecutive series of questionnaires.Reference Keeney, Hasson and McKenna 24 In this formative study, expertise was defined by participants having either (1) a peer-reviewed publication within the area of disaster management or (2) disaster management experience. This formative study was designed to target a broad audience of disaster management professionals and was not designed to be internationally representative. A broad audience was targeted to generate a robust set of factors and foster increased participation in this formative study, as there is no universally agreed upon criteria for the selection and number of peer reviewers required for a Delphi study.Reference Keeney, Hasson and McKenna 25
To identify possible factors, a literature review was conducted by using the UC Irvine Libraries from the University of California. The search term “mass casualty incident” with no date limit and the “Libraries Worldwide” selection was used and yielded 2955 resources (ie, videos, e-books, books, and articles). Within these results, the search term “distribution of patients during a mass casualty” returned 85 resources. Criteria for selection of relevant resources included resources that identified factors relating to the distribution of patients during a mass casualty incident. Upon review, 40 resources were selected.Reference Peleg and Kellerman 1 - Reference Mackaway-Jones 4 , Reference Levi, Bregman and Geva 19 , Reference Abend, Bubke and Hotop 26 - Reference Welling, Boers and Mackie 60 In addition, gray literature such as government technical and working group reports were also incorporated. 61 - 72
From the selected 40 resources, these resources generated a total of 996 factors that when themed resulted in 56 unique factors related to the distribution of patients in a mass casualty incident. Qualitative thematic analysis was used to theme the 56 unique identified factors and these factors served as the foundation questions for the round one questionnaire.Reference Ryan and Bernard 73
Using purposive sampling, round one began with an invitation sent via e-mail to 40 disaster management professionals who were thought to meet the criteria of participation asking them to participate in two online questionnaires within a 30-day period and to forward the research request to other appropriate colleagues. To complete the round one questionnaire, peer reviewers were asked to rate how much influence, using a 9-point Likert scale, each of the 56 identified factors would have on their decision-making related to the distribution of patients during a mass casualty incident. The peer reviewers were also asked to identify any additional factors of influence that were not included in the round one questionnaire. The online questionnaire tool Survey Monkey 74 was used to collect the data.
Once questionnaire results were collected, the degree of agreement was determined for each factor. The criteria for agreement were defined as a median score greater than or equal to 7 (on a 9-point Likert scale), and a percentage distribution of 75% or greater of ratings being in the highest tertile (ie, 7-9).Reference Keeney, Hasson and McKenna 25 , Reference Boulkedid, Abdoul and Loustau 75 - Reference Alahlafi and Burge 77 The criteria were not shared with peer reviewers to reduce the possibility of the “anchoring effect.”Reference Wilson, Houston and Etling 78
Factors that met the criteria for agreement from the round one questionnaire were considered accepted and removed from the round two questionnaire. The round two questionnaire included those factors that did not meet the defined criteria for agreement from the round one questionnaire plus any newly identified factors from the round one questionnaire. The newly identified factors identified by peer reviewers in round one were included in round two if they were new themes from the factors identified in round one. For the round two questionnaire, peer reviewers were provided with information that included their individual response from the round one questionnaire and the group level dispersion (ie, first, second, and third quartile). The peer reviewers were asked to reconsider their original response in light of the group response, rating again how much influence each factor would have on their decision-making related to the distribution of patients during a mass casualty incident.
Microsoft Excel 2011 version number 14.3.6 (Microsoft Corporation) was used to analyze and manipulate the data. Research Ethics Board review was obtained from Sunnybrook Health Sciences Center, an academic, university-affiliated hospital in Toronto, Canada.
RESULTS
Forty invitations were sent requesting participation in this formative study, from which 17 peer reviewers completed the round one questionnaire. An additional 45 peer reviewers were identified through referral. In total, 62 peer reviewers completed the round one questionnaire, and 54 (87%) of them also completed the round two questionnaire. Of the 54 peer reviewers who responded to both questionnaires, 13 countries were represented with the largest group being 35 peer reviewers (65%) from Canada (Supplemental Table 1 in the online data supplement). The 2 most represented practice settings were hospital (39%) and prehospital (31%) settings (Supplemental Table 2 in the online data supplement). Twenty peer reviewers (37%) had greater than 15 years of disaster management experience. Twenty-five (46%) had attended 5 or more mass casualty incidents, and 47 (87%) had attended at least one mass casualty incident (Supplemental Table 3 in the online data supplement).
Eighteen of the 56 factors (32%) that were identified through the literature review met the criteria for agreement in the round one questionnaire (Table 1). The participating peer reviewers identified an additional 8 factors to be included in the round two questionnaire. A total of 46 factors were therefore presented in the round two questionnaire, and 11 (24%) of these factors met the criteria for agreement (Table 2). The remaining 35 factors from the round two questionnaire did not meet the criteria for agreement (Table 3).
Table 1 Round One Questionnaire Factor Agreement (n=62)Footnote a
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20180323110339277-0413:S193578931700043X:S193578931700043X_tab1.gif?pub-status=live)
a Abbreviations: CBRN, chemical, biological, radiological-nuclear; Q, quartile.
Table 3 Factors That Did Not Meet Agreement After the Round Two Questionnaire (n=54)Footnote a
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20180323110339277-0413:S193578931700043X:S193578931700043X_tab3.gif?pub-status=live)
a Abbreviations: PPE, personal protective equipment; Q, quartile.
DISCUSSION
It was defined a priori that agreement was considered to be met if the median was a rating of 7 or greater and if the total distribution of the responses was greater than 75% in the highest tertile (ie, 7-9). Of the 64 factors identified in this modified Delphi study, 29 factors (45%) met the criteria for agreement upon completion of the round two questionnaire. Overall, the degree of agreement may suggest that certain factors are influential to decision-making related to the distribution of patients in a mass casualty incident.
Factors That Met the Criteria for Agreement
Of the 29 accepted factors, 12 (41%) were related to incident details and 5 (17%) were related to resources. This verifies that incident details and availability of resources are influential to decision-making related to the distribution of patients during a mass casualty incident. To assist with assessing incident need, a checklist of the accepted factors related to incident details and resources could serve as a tool or job aid for error management and performance improvement for mass casualty incidents (ie, a tool to prompt performance and safety measures).Reference Van de Walle and Turoff 79
Furthermore, of the 29 agreed upon factors, 12 (41%) were related to systems such as incident management, communication, continuity of operations, policies, and procedures.Reference Levi, Bregman and Geva 19 , Reference Ficke, Johnson and Hsu 43 , Reference Welling, Boers and Mackie 60 This suggests that these system factors are susceptible to influence through proactive system planning, testing, and risk identification (ie, developing standard procedures for mass casualty incidents). The high degree of agreement related to system factors is important as it suggests an opportunity to decrease variability within the response and management of mass casualty incidents. Agreement on the factor identification and analysis of potential hazards and risks further illustrates the importance of proactive identification and mitigation of risk prior to actual incidents.
Factors That Did Not Meet the Criteria for Agreement
Although agreement was not met on a total of 35 factors, it is valuable to analyze where there was potential for agreement and more importantly where there was difference within the responses.
The least amount of agreement of the 35 factors that did not reach agreement was found on the following factors: associated costs of response, forensics, patient characteristics, and coordination of resources by a single national authority. The finding that the factor coordination of resources by a single national authority was not influential is in contrast to the Israeli surge capacity essential tasks that seeks coordination of resources nationally.Reference Peleg and Kellerman 1 An explanation for this finding may be the large proportion of Canadians within the sample of peer reviewers (65%) and the challenges to nationally centralize resources due to the geographical boundaries (ie, land size) and political structure in this country. It is important to note that cost was not determined to be an influential factor nor was forensics, which may suggest that potential lives saved are perceived to be more valuable than fiscal restraint and preservation of evidence.
Conducting a subanalysis of only those peer reviewers that identified their practice setting as hospital (n=21) and prehospital (n=17) for the 35 factors that did not reach agreement illustrated that agreement would have been met for 5 factors within the hospital setting and 13 factors within the prehospital setting (Table 4). From the abovementioned subanalysis, the following factors would have met the criteria for agreement for both the hospital and prehospital settings: inventory control and supply chain planning, incident characteristics, level of incident isolation, and standard procedures for knowledge translation. These factors are operational in nature and focus on response incident details and preparedness. This suggests that incident details (ie, incident characteristics such as event type, scope, scale) may influence the decision-making related to the distribution of patients in a mass casualty incident, particularly for first responders (ie, prehospital) and first receivers (ie, hospital). Specifically, “type of disaster” is noted as a prominent factor in first responder acronyms (ie, METHANE, CHALETS, and HANE)Reference Mackaway-Jones 4 and in the literature.Reference Peleg, Michaelson and Shapira 3 , Reference Abend, Bubke and Hotop 26 , Reference Brandeau, McCoy and Hupert 35 , Reference Hupert, Mushlin and Callahan 48 , Reference Murray and Goodfellow 53 , 71 Furthermore, the factor standard procedures for knowledge translation (ie, education, technical training, exercises) is consistent with the World Health Organization’s Hospital Emergency Response Checklist and knowledge translation is further noted in the literature as a prominent factor.Reference Timbie, Ringel and Fox 2 , Reference Mackaway-Jones 4 , Reference Christie and Levary 40 , Reference Hupert, Wattson and Cuomo 49 , Reference Phelps 55 , Reference Sanders 57 , 67 , 71
Table 4 Subanalysis of the 35 Factors That Did Not Reach Agreement for Only Those Experts That Identified Their Clinical Setting as Hospital (n=21) and Prehospital (n=17)Footnote a
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20180323110339277-0413:S193578931700043X:S193578931700043X_tab4.gif?pub-status=live)
a Abbreviation: Q, quartile.
Limitations
The modified Delphi approach has limitations that should be taken into consideration. One limitation of the modified Delphi approach is the subjectivity in the definition of “expert” as defined as a peer reviewer.Reference Keeney, Hasson and McKenna 24 , 67 In this study, the definition of a peer reviewer was designed to target a broad audience of disaster management professionals in order to generate a robust set of factors and foster increased participation. To mitigate excessive diversity among peer reviewers, recruitment was purposive and original invitations to participate in this formative study were sent out to disaster management professionals who were thought to have met the criteria for participation and by peer reviewer referral. This purposive sampling approach may serve as a limitation of this study as peer reviewers are from similar networks and was not designed to be internationally representative. Another potential limitation of the modified Delphi approach is subjectivity in the definition of the criteria for agreement.Reference Keeney, Hasson and McKenna 24 In order to minimize this limitation, the criteria for agreement incorporated both percentage distribution (ie, 75% or greater of ratings being in the highest tertile) as well as median score (ie, median greater than or equal to 7), as opposed to other Delphi studies that only measure percentage distribution.Reference Campbell, Cantrill and Roberts 76
Additional limitations of the modified Delphi approach include that it does not allow peer reviewers to expand on their view as they would in a face-to-face setting. However, this may also be considered a strength in this study, as it may help to mitigate the “bandwagon” effect.Reference Keeney, Hasson and McKenna 24 Also, merely because a factor met the criteria for agreement in this study does not mean that wide consensus exists or that it is the correct answer.Reference Keeney, Hasson and McKenna 24 In terms of generalizability, the results should be interpreted with caution owing to the large proportion of Canadian respondents (65%) and the fact that this formative study was not designed to be internationally representative. Finally, due to the complex and uncertain nature of mass casualty incidents, it is not known how well the identified influential factors in this study would perform in the field, nor whether participating peer reviewers would respond differently in a true mass casualty incident than they indicated through their responses in this study.
Future Research
A suggestion for future research may be to develop decision-support tools (eg, job aids) to assist decision-making for first responders and receivers in assisting with the distribution of patients during a mass casualty incident. These tools could be created by using the factors that achieved the highest levels of agreement as found in this formative study. Furthermore, computational modeling using the factors that achieved agreement in this study may assist planners and policy-makers in determining and testing distribution options for patients during a mass casualty incident.
CONCLUSIONS
Fifty-four disaster management peer reviewers assessed factors that may influence decision-making related to the distribution of patients during a mass casualty incident. More than 86% of the peer reviewers agreed that the number of estimated and actual casualties was influential to decision-making, as were the availability of transportation, understanding injury characteristics, and whether the patient had any special considerations such as burn, ophthalmologic, neurology, pediatric, etc, and that reducing risk through proactive system planning is influential, through the creation of standard procedures for mass casualty situations. Overall, the degree of agreement suggests that certain factors are influential to decision-making related to the distribution of casualties in a mass casualty incident. Practically, the factors that achieved the highest levels of agreement as identified in this formative study could be used to create decision-support tools (ie, job aids) to assist with connecting and protecting first responders and receivers. These tools, guided by the accepted factors, may assist with developing principles for distribution of patients and may assist planners and policy-makers with standardizing response processes and plans. Furthermore, the additional factors that did not reach agreement may serve as a mechanism for discussion and further assist with planning and education. The agreed upon factors may decrease variability within the response and management of mass casualty incidents.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/dmp.2017.43