Introduction
The Comprehensive Framework for Disaster Evaluation Typologies,Reference Wong, Spencer, Boyd, Burkle and Archer1 developed in 2017 (CFDET 2017), unifies and facilitates agreement on the identification, structure, and relationships between the various evaluation typologies found in the disaster setting. Despite the evolution of disaster evaluation and research, and concerted efforts since 2003 to develop consensus on disaster research and terminology, 2–Reference Stratton20 little agreement from experts exists for a comprehensive and validated framework that structures disaster evaluation typologies. Previous frameworks, 2–Reference Stephenson5,Reference Glassey8–Reference Birnbaum, Daily, O’Rourke and Loretti18 while recognized in the literature, lacked validation and were rarely used as a methodological framework to structure disaster evaluation and research. Reference Wong, Spencer, Boyd, Burkle and Archer1 To add credibility and global relevancy to CFDET 2017, an international peer-reviewed validation process was conducted. Reference Cardoni21 The primary purpose of this paper discusses the validation process, including results and outcomes.
Background
The CFDET 2017 was developed around two key elements: the different phases of disasters (also referred to as the disaster timeline), Reference Government22–Reference Neal27 and the evaluation typologies that have been or could be used in the disaster setting. The extensive, underpinning literature review of evaluation typologies previously reported Reference Wong, Spencer, Boyd, Burkle and Archer1 were categorized into the following elements:
Core Structure;
Baseline Evaluation Typologies;
Consequence Evaluation Typologies;
Outcome Evaluation Typologies;
Impact Evaluations;
Accountability; and
Evaluation Standards, Evidence, and Knowledge Management.
While Accountability and Evaluation Standards, Evidence, and Knowledge Management are not true evaluation types, as recognized by the evaluation sector, their importance to the process 28,29 was deemed significant enough for inclusion in CFDET 2017.
Each element was brought together into a single, unifying framework: CFDET 2017. The full development and rationale behind the structure of CFDET 2017 (Figure 1), including the scoping literature review, was published in Prehospital and Disaster Medicine (PDM) in 2017. Reference Wong, Spencer, Boyd, Burkle and Archer1
Figure 1. Comprehensive Framework for Disaster Evaluation Typologies 2017.
Abbreviations: GFDRR, Global Facility for Disaster Reduction and Recovery; GIS, Geographic Information Systems; HIA, Health Impact Assessment; PDNA, Post-Disaster Needs Assessment.
The CFDET 2017 promotes an environment for constructive dialogue on evaluation typologies currently in use, or that could be used, in the disaster setting, and used as a tool to support decision making when undertaking evaluations before, during, or after a disaster. Furthermore, it provides structure to improve the selection and reporting of valid disaster evaluation and research typologies, thereby strengthening the evidence-base for interventions delivered across the disaster spectrum.
This study addresses the research problem that previous frameworks, identified in the literature, lacked validation and consistent terminology. To gain credibility and utility, this unique framework, CFDET 2017, needed to be validated by international experts in the disaster setting.
Validation Process
A focused literature review, including peer-reviewed and grey literature, identified articles related to the validation of “comprehensive frameworks,” similar to CFDET 2017. Such articles were limited in the disaster or evaluation settings. The literature highlighted multiple perspectives and confusing terms and typologies for “validation” and how to undertake a “validation process.” Reference Creswell and Miller30 This paper defines “validity” as: “the ability of the instrument to measure the attributes of the construct under study.” Reference DeVon, Block and Moyle-Wright31 In this paper, the instrument under study is CFDET 2017, including its “attributes” that are the concepts and elements (disaster phases and evaluation typologies found in the disaster setting) and their relationships. Validity can be further divided into “external validity,” which measures the generalizability of the findings, and “internal validity,” which refers to the confidence placed in the cause and effect relationship. Reference Keeney, Hasson and McKenna32 To make a framework useful, it needs both internal and external validity. Reference Fattah, Rehn, Reierth and Wisborg33
The literature review identified no gold standard guidelines or consensus for undertaking a validation of CFDET 2017. The literature identified a wide variety of validation practices ranging from complex three-step Delphi processes to opinion reviews, by sometimes as few as two experts. Reference Argyrous34–Reference Jeon, Conway, Chenoweth, Weise, Thomas and Williams38
Davis-Stober, et al Reference Davis-Stober35 and Jorm Reference Jorm36 independently reviewed the “wisdom-of-the-crowd” effect. Jorm argues that although evidence-based medicine has gained strength over the past two decades, especially for informing clinical decisions, and despite “expert consensus” rating lowly on the Levels of Evidence Scale, “expert evidence” should not automatically be classed as an inferior method. Reference Jorm36 To support his argument, Jorm summarized the literature on Surowiecki’s concept of the “wisdom-of-the-crowd,” where the term “crowd” referred to “any collection of individuals with some expertise.” Jorm quotes Surowiecki who proposed that certain conditions must be met for a “crowd to be wise” and for expert consensus to produce good answers; namely: (1) diversity of expertise; (2) independence: the expert panel members make their decisions independently, uninfluenced by others; (3) decentralization of panel members; and (4) a mechanism of coordinating and aggregating the crowd’s expertise. Reference Jorm36,Reference Surowiecki37 Davis-Stober, et al also modelled “wisdom-of-the-crowd” effect with a mathematical approach and concluded that: “While the members are individually biased and the crowd not particularly accurate, the crowd is still wise relative to the individual.” They also supported the importance of two of Surowiecki’s conditions, namely: diversity of the crowd; and aggregation of the crowd’s expertise. Reference Davis-Stober35
The CFDET 2017 validation process primarily aimed to capture the “wisdom-of-the-crowd” represented by a wide group of international experts. Surowiecki’s conditions guided the CDFET 2017 validation methodology.
The first challenge was to pre-determine the size and nature of the crowd, or expert panel. Jorm, in reviewing the “wisdom-of-the-crowd” literature, specifically focusing on determining expert panel size in Delphi studies, concluded that there was little firm guidance and that the stability of responses was demonstrated with panels of around 20 members. Reference Jorm36 The CFDET 2017 validation aimed for 30 responses. Jorm also advised that panel attrition in Delphi studies may be larger if the questionnaire is long or requires a substantial time commitment. Reference Jorm36 An observation that proved prophetic in this validation study.
The second challenge was to pre-determine the consensus thresholds. Jorm also reviewed the quantitative definition of “consensus” used in Delphi or consensus studies, concluding that no single definition existed for consensus and researchers would need to define consensus and provide a rationale in the context of their own studies. The definition for consensus varied between 70% and 90% to determine endorsement as “essential or important.” Reference Jorm36 For this CFDET 2017 validation, four definitions of consensus thresholds were used and are defined in the Methods section of this paper. Reference Jorm36,Reference Jeon, Conway, Chenoweth, Weise, Thomas and Williams38
Methods
A mixed methods approach validated the comprehensive framework. A preliminary iterative process of presenting CFDET 2017 at international conferences, national seminars, and other academic meetings provided opportunities to discuss and receive formative feedback on the evolving CFDET 2017 prior to undertaking the summative validation process. This approach embraced the underlying principle of Action Research as an iterative process seeking input from end-users. Reference Ivankova and Wingo39 Evolving versions were presented and discussed at three international conferences, including: the World Congress on Disaster and Emergency Medicine (World Association for Disaster and Emergency Medicine [WADEM]; Madison, Wisconsin USA), Cape Town, South Africa, April 2015; the United Nations Office for Disaster Risk Reduction (UNISDR; Geneva, Switzerland) Science and Technology Conference, Geneva, Switzerland, January 2016; and the World Library and Information Congress (International Federation of Library Associations [IFLA]; The Hague, Netherlands), Kuala Lumpur, Malaysia, August 2018; as well as at three further professional and research forums at Monash University (Melbourne, Australia). Feedback received contributed to progressive modifications to the evolving CFDET 2017; for example, the addition of arrows to indicate feedback processes and “complete the circle” of information. Other comments highlighted an opportunity to further clarify the theory underpinning the structure. These iterative events and the feedback received strengthened the development of CFDET 2017 and were a key component of the progressive validation process.
The formal validation process used an online survey, hosted on the Qualtrics Insight Platform (Qualtrics; Provo, Utah and Seattle, Washington USA),40 to collect quantitative (five-point Likert Scale) and qualitative data. An online survey, rather than interviews or focus groups, was selected due to the intended sampling size and the international nature of the participants. Reference Jorm36 Given that participants were provided with an overview of CFDET 2017 as the instrument under study, which included the key concepts and elements, the survey was intended as a one-step survey. There was the potential to use a second-round survey had the first-round survey failed to reach consensus; however, this was not necessary as the first survey attained consensus.
A questionnaire was developed and pilot tested (n = 9). Results indicated no major structural changes or modifications were required; however, some minor edits were made to the survey instrument. Human Research Ethics approval was obtained from Monash University.
One hundred and forty experts with backgrounds in the areas of: disaster medicine and disaster/emergency management; humanitarian/development; or evaluation generalists, were approached to participate in the validation via email. Both non-probability and purposive sampling were used to recruit the target population. Reference Bamberger, Rugh and Mabry41,Reference Markiewicz and Patrick42 The stratification of participants was based on the following two characteristics of importance: primary country of residence with participants allocated into one of the six World Health Organization (WHO; Geneva, Switzerland) Regions of the World; 43 and current field of experience (participants self-selected one of the following three sub-groups: disaster medicine, disaster management, and emergency management sectors; humanitarian and development sectors; or evaluation sector).
The recruitment list was sourced from: the literature; the public domain; and known professionals active in the fields of disasters/emergencies, humanitarian/development, and/or evaluation. An explanatory statement was sent via email, and consent to participate in the research was requested. Participants had three weeks to complete the survey with a reminder email sent automatically at two weeks. Recruitment used a snowballing strategy until the consensus threshold of 30 responses was reached. The survey ran for a six-month period in 2017.
The online survey consisted of two elements: (1) an overview, and (2) an explanation of CFDET 2017, which was consistent with the 2017 publication Disaster Metrics: A Comprehensive Framework for Disaster Evaluation Typologies,Reference Wong, Spencer, Boyd, Burkle and Archer1 although it was “in print” and had not yet been published. Participants were asked to:
Confirm their consent to participate and respect the intellectual property contained in CFDET 2017;
Complete a short section on their demographics;
Use a five-point Likert Scale (five = strongly agree, one = strongly disagree) to identify their agreement or disagreement for a series of statements related to the elements and relationships found within CFDET 2017;
Provide responses to open-ended questions, with opportunity to comment on the disaster timeline elements and the different evaluation typologies presented;
Comment on the relationships and linkages between the disaster timeline and evaluation typologies as demonstrated in CFDET 2017, whether CFDET 2017 had value when undertaking evaluations, would be useful for supporting and promoting evaluations, and whether it would be useful for teaching evaluation in the disaster setting; and
Suggest other evaluation typologies that were or could be used in the disaster setting but were not included in CFDET 2017, identify other improvements to CFDET 2017, and identify potential barriers and enablers to undertaking evaluations in this setting.
Analysis of the quantitative responses used definitions and descriptive statistics facilitated by a Statistics Canada Platform (Statistics Canada; Ottawa, Canada)44 and the Qualtrics Insight Platform. 40 Qualitative responses utilized a thematic analysis facilitated by an Excel spreadsheet (Microsoft Corp.; Redmond, Washington USA). As no single definition of consensus exists,Reference Jorm36 a pre-survey determination was made that the following criteria would be used to indicate acceptance thresholds for the validation of CFDET 2017:
Total of n = 30 responses;
Mean of 4.00, + /- 95% Confidence Intervals (CI), on a five-point Likert Scale for each item;
Median of 4.00, + interquartile range, on a five-point Likert Scale for each item; and
Total of 80% of participants who “strongly agree” or “somewhat agree” on a five-point Likert Scale for each item.
Results
Demographics of Responders
Of the 140 experts approached, 39 (27.9%) completed the demographic section; however, only 33 (23.6%) completed all, or most, of the questionnaire related to CFDET 2017.
Demographic information showed that the number of female-to-male participation was almost equal, with 51.3% and 48.7%, respectively. The majority of participants were aged 41 years and above (92.3%), with 33.3% greater than 60 years. While participants were invited from all six WHO Regions, responses were only received from four regions: Western Pacific (n = 22), European (n = 11), Americas (n = 5), and Eastern Mediterranean (n = 1). However, 11 countries were represented. Nearly two-thirds of participants (64.1%) identified their area of expertise as disaster medicine, disaster management, or emergency management; 28.2% identified evaluation; with the remaining participants (7.7%) categorizing themselves as belonging to the humanitarian or development sectors. Two-thirds of participants (66.7%) had greater than 11 years or more of full-time experience in their self-selected area of expertise, and of those, 46.1% had 16 (or more) years of experience. The majority of participants (64.1%) indicated that they undertook evaluations or supervised evaluations as part of their normal work.
Elements and Relationships Represented within CFDET 2017
Table 1 provides the results for the responses on the elements and relationships represented with the framework.
Table 1. Elements and Relationships Represented Within CFDET 2017
Abbreviation: CFDET, Comprehensive Framework for Disaster Evaluation Typologies.
Of the seven CFDET 2017 elements, the results show:
All had between 29 and 33 responses;
Four elements had a mean of > 4.00. The remaining three elements had means close to 4.00, and all had 95% Confidence Interval (CI), which included 4.00;
All had medians = or > 4.00; however, the interquartile range indicated the responses were skewed to the right, as may have been expected; and
Four elements received > 80% “strongly agree” or “somewhat agree,” while three elements did not meet this consensus threshold.
Explanations for these observations are potentially apparent in the qualitative responses. In the qualitative responses, a total of 114 comments were specifically related to the elements within the framework. There were no strongly negative comments or suggestions for major change, except related to Accountability. Some clarification suggestions were offered for the other elements and are considered in the Discussion section of this paper.
Holistic Representation as a Unifying Framework
Table 2 provides the results for the responses on the holistic representation as a unifying framework. Of the four questions relating to the holistic representation as a unifying framework, the results show:
All had at least 29 responses;
All exceeded a mean of > 4.00;
All had a median of = or > 4.00; however, the interquartile range indicated the responses were skewed to the right, as may have been expected; and
All comfortably received > 80% “strongly agree” or “somewhat agree.”
Table 2. Holistic Representation as a Unifying Framework
Abbreviation: CFDET, Comprehensive Framework for Disaster Evaluation Typologies.
Respondents were asked to rate and comment on whether the relationships between the disaster timeline and evaluation typologies were appropriately demonstrated within CFDET 2017. Quantitative results met all four consensus thresholds (Table 2). Qualitative comments endorsed CFDET 2017 with the following being typical comments: “Makes sense for the way it is illustrated;” “Great work putting it altogether in one diagram, it’s intuitive, well done;” “Relationships are appropriately demonstrated;” and “I think this final framework is excellent.”
Respondents were asked to rate and comment on whether CFDET 2017 potentially held value when undertaking evaluations in the disaster or humanitarian setting. While the number of responses was slightly lower than preferred (n = 29), quantitative results met the other three consensus thresholds (Table 2). Qualitative comments endorsed CFDET 2017 with the following being typical comments: “I think this can inform different understandings of the range of not just evaluation, but sources of data to analyze and inform better service delivery in humanitarian contexts;” “There is a need to show evaluation as a continuum – which your framework shows;” “It helps to understand the wider context and provides insight into how current and future studies relate to the overall picture;” “The various elements are clearly linked to the various phases of disaster;” “Clear, thoughtful overview, higher level approach provides a useful framework;” and “The framework provides clear visual representation.”
Respondents were asked to rate and comment on whether CFDET 2017 would be useful for supporting and promoting evaluations in the disaster or humanitarian setting. While the number of responses was slightly lower than preferred (n = 29), quantitative results met the other three consensus thresholds (Table 2). Qualitative comments endorsed CFDET 2017 with the following being a typical comment: “As clearly explained in the introduction of this survey, there is not a clear standard and framework for disaster evaluation. Such as well-developed systematic framework would definitely help in supporting and promoting sound disaster evaluation.” Further comments included: “It synthesizes the disaster evaluation process;” “The framework is a useful visual representation that can be used to support and promote evaluations;” “Identifies the facts to evaluate/measure;” “Evaluators, particularly workers who are researching as a result of an event and are not academics, would benefit from a clear framework to base their evaluation on;” and “By providing an internationally agreed upon framework, it has the benefit of providing guidance on the research and clearer understanding of responsibilities in this field.”
Respondents were asked to rate and comment on whether CFDET 2017, as a unifying framework, would be useful for teaching evaluation in the disaster or humanitarian setting. Quantitative results met all four consensus thresholds (Table 2). Qualitative comments endorsed CFDET 2017 with the following being typical comments: “Each element is clearly linked to the various phases of disaster and provided a very useful visual representation, that is not only useful for garnering support but also for pulling apart in teaching environment;” “The framework graphically aids students learning the complexity and importance of relations among all elements;” “Breaks down the system to simpler parts;” “Clean and organized;” “Helps to understand the process;” “Provides a clear representation;” and “[A] very logical step-by-step framework.”
Additional Questions
Respondents were asked to identify any typologies for each of the elements that may have not been included in CFDET 2017. A number of suggestions were offered; however, most of these had been included in the published version of CFDET 2017, which was not available to respondents at the time of the survey.
Respondents were asked to offer their thoughts on improving CFDET 2017. Only 11 additional comments were offered. No comments related to the structure of CFDET 2017. Most comments related to operationalizing the framework and included: “The best way for improvement is by testing the framework on the ground;” “My suggestion is not to suggest it replaced all other forms of organizing ideas in this area of work, just suggest it adds to and pulls together many other elements;” “Please be bold enough to revise once this work is out and has been utilized for a certain period of time;” and “You could make many sub-notes for each section of the diagram to clarify what each section means; the diagrams should be easily understood with no knowledge of the field.”
Respondents were invited to comment on any barriers and enablers they perceived in undertaking evaluations in this setting. Only 13 responders commented on barriers and eight commented on enablers. Summative comments on barriers included:
“Complexity of evaluation guidelines” (3), and “Their difficulty in being operationalized” (2);
“Lack of consistency of definitions, criteria, standards, and ‘on-the-ground’ tools” (3);
“Poor understanding of evaluations, lack of experience/training” (2);
“Disaster setting is complex and sophisticated and is the first barrier to undertaking evaluations;”
“Accessibility and consistency of data;”
“Ethics;”
“Short-term funding;”
“Expectations of donors;” and
“Evaluations are not seen as priorities among emergency management/humanitarian workers.”
Summated comments on enablers included:
“Clear guidance for evaluators; if we use the same framework, we can compare outcomes; recognize the generic utility and potential for comparisons that are apparent with a process such as this” (3);
“Enthusiasm; capacity building, increasing awareness, and sharing knowledge are key elements for better application of these evaluations” (2);
“Early publications of evaluations linked to the framework” (2);
“Be pro-active, maintain good relationships, and build up confidence with the right people at ministries and sponsor organizations;” and
“The use of a national framework will help promote the various elements and enhance understanding of the need to undertake evaluations in the various settings suggested by the framework.”
The stated aim of the comprehensive framework addresses many of these barriers and uses many of these enablers to move forward.
Discussion
Of the international experts who responded to the peer-reviewed validation survey, there was overwhelming support and endorsement for the concept, structure, and usefulness of CFDET 2017. The survey provided a valuable opportunity and independent input to improve the comprehensive framework.
Core Structure
The Core Structure of CFDET 2017 consisted of three layers: first, a simplified disaster timeline including stages of pre-event, event, and post-event; second, a more detailed sequence of the disaster timeline identifying: pre-event status; hazard; risk reduction; event; damage; disruption, change in functions, and consequences; response; recovery; and post-event status; as well as a third layer, strengthening resilience. Reference Wong, Spencer, Boyd, Burkle and Archer1
Quantitative results on Core Structure met all four consensus thresholds (Table 1).
Qualitative responses provided two respondents’ comments influencing modifications to the Core Structure. The first was that “rehabilitation” be included within the recovery section and was accepted. The Sendai Framework for Disaster Risk Reduction 2015-2030 (SFDRR) mentions rehabilitation as a major priority (Priority 4: Enhancing Disaster Preparedness for Effective Response). 45 The SFDRR defines rehabilitation as: “A set of measures aimed at restoring normal living conditions through the repair and re-establishment of vital services interrupted or degraded by a disaster or emergency.” 46 Importantly, recovery actions can occur concurrently and are not mutually exclusive. Reference Coppola47
The second comment recommended the addition of the International Health Regulations (IHR; 2005) 48 into the strengthening resilience section. This was a valid suggestion, as one of the underlying themes of CFDET 2017 includes health. The IHR is a legally binding international instrument for the 196 signatory countries, including Member States of the WHO. The IHR aims to help international communities prevent and respond to acute public health risks that have the potential to cross borders and threaten the health of people world-wide. 48,49
Two further international consensus frameworks were deemed globally significant and were included: The World Humanitarian Summit (WHS; 2016) 50 and the Urban Conference on Housing and Sustainable Urban Development (Habitat III; 2016). 51
All guidelines and frameworks included in the Core Structure reflect international significance, and their inclusion in the updated Core Structure demonstrates the contemporary nature and living status of the modified framework. Although not all are legally binding, these international consensus frameworks have shaped policy and practice globally and add rigor and strength to CFDET 2018.
Baseline Evaluation Typologies
While the number of responses was slightly lower than preferred (n = 29), quantitative results met the other three consensus thresholds (Table 1). Qualitative comments provided insight that the inclusion of pre-event hazard and risk reduction strategies as Baseline Evaluations may have caused some confusion. Recent examples of Baseline Evaluations, including the new Australian Vulnerability Profile, confirm the decision to retain this element in the comprehensive framework. Reference O’Connell, Wise and Doerr52,53 Baselines Evaluations and baseline assessments remain an integral part of disaster risk reduction as they provide a critical reference point for assessing changes and impact. 54 Furthermore, they establish a basis for comparing the community situation before and after a disaster. The inclusion of the word “assessments” is reflective of a survey response and intended to provide further clarity within the Baselines section. The term “evaluability” in Baseline Evaluations appears to have been misunderstood. Evaluability is a pre-evaluation assessment that the program or intervention intended for evaluation “is able to be evaluated” (eg, are there adequate data and are stakeholders available and willing to participate in the evaluation?). However, it is not an evaluation in itself and was removed from CFDET 2017 as a stand-alone entity.
The importance of common statistical and operational datasets was also mentioned in a responder’s comment. Reflecting this response, changes to demographics and infrastructure were made to include “existing common statistical and operational databases.” The term “databases” was preferred and used instead of the term “datasets” to ensure the recognition of broader data sources. Common Operational Datasets (CODs) and Fundamental Operational Datasets (FODs) provide important databases in humanitarian emergencies. They can be used to improve the effectiveness of humanitarian assistance by supporting technical standards, improving the quality of the data, and strengthening inter-operability and harmonization as developed and endorsed by the Inter-Agency Standing Committee (IASC; Geneva, Switzerland). 55–57
A recent environmental scan of the literature identified the expansion of the use of targets and indicators in this domain. The CFDET 2017 Baseline Evaluations already included “Define Minimum Standards Criteria: Baselines, Targets, and Indicators;” however, to reinforce the importance of this typology and in recognition of the evolution of the targets and indicators developed and endorsed by the United Nations after the Sendai Conference, 58 an opportunity was taken to make an additional change to the comprehensive framework. The emergence of targets and indicators reflects the maturing of the disaster evaluation approach. Significantly, targets and indicators will be used every two years as the framework for all countries to report on their progress towards disaster risk reduction goals within the SFDRR. 45 The updated Baseline Typologies now includes explicit baseline targets and indicators, and the disaster risk reduction targets and indicators of the SFDRR.
Subsequent to and within the principles of the SFDRR, the WHO led a global approach to Health Emergency and Disaster Risk Management (H-EDRM),59 which complements the SFDRR. This evolving WHO framework includes a thematic platform, research agenda, and technical guidelines, and also has been included in the Baseline Typologies.
The final modification made to the Baseline Evaluation Typologies section included the title Baseline Typologies as a descriptor, which provides clarification and direction to the items included within.
Consequence Evaluation Typologies
Quantitative results on Consequence Evaluations met all four consensus thresholds (Table 1). Qualitative responses suggested two modifications. The timeframes mentioned in the original rapid needs assessment and detailed needs assessment appeared to be taken literally and generated discussion. The timeframes were intended to be indicative only, and in acknowledgement of the feedback, the term “For example (eg)” was added to the timeframes mentioned under rapid needs assessment (damage): eg, Day 1 and Detailed Needs Assessment (Functional); eg, Days 2-3: These timeframes are also as per the IASC’s guiding document on Initial Rapid Assessments. 60
The second modification made to the Consequence Evaluations section included the title “Consequences Typologies” as a descriptor, which provides clarification and direction to the items included within.
Outcome Evaluation Typologies
Quantitative results on Outcome Evaluations met all four consensus thresholds (Table 1). Qualitative responses provided suggested two minor modifications. Firstly, the descriptor “Outcome Typologies” was added to provide further clarification and direction to the items included within Outcome Evaluations. Secondly, Accountability, which was shown as a stand-alone element in CFDET 2017, was suggested as being better located in the Outcome Evaluations section, and was thus relocated. The rationale is outlined in detail in the Discussion section on Accountability.
Reflecting the contemporary nature of the comprehensive framework, the opportunity was taken to include “recovery indicators” in the Outcome Typologies section. This new typology has recently emerged and is used in the recovery phase of the disaster timeline. Recent literature suggests an attempt to identify factors that may lead to effective or good recovery and strategies to monitor the community’s progress towards recovery after major events. 61,62 Further consideration is included later in this Discussion section.
Impact Evaluations
Quantitative results on Impact Evaluations only met two of the four consensus thresholds (Table 1). There was an expectation that the inclusion of Impact Evaluations in CFDET 2017 would generate discussion, and this proved to be true. Qualitative respondents offered different interpretations and definitions related to Impact Evaluations. Contention and disagreement regarding definitions and terminology was supported by the responses received: “Most of the available impact evaluations of humanitarian action are deeply flawed: first, there is the problem of attribution, and second is the problem of mechanism.”
Qualitative responses suggested a number of respondents were unaware of the contemporary nature of Impact Evaluations as evidenced by one exemplar comment: “I think causal link assessment is closely linked with inquiry style assessments; in the pre-event stages, this would probably be referred to as a risk assessment.”
Despite on-going definitional debates,Reference Veenema63 Impact Evaluations are particularly well-suited to answer important questions, such as: whether interventions do or do not work; whether interventions make a positive or negative impact; whether there are intended or unintended consequences; and how cost effective they are. Reference Bamberger64,Reference Puri, Aladysheva, Iversen, Ghorpade and Bruck65 Correctly structured and implemented Impact Evaluations are expected to greatly improve the effectiveness of interventions delivered in the disaster setting by identifying what works for whom and why. 66,Reference Puri, Aladysheva, Iverson, Ghorpade and Bruck67
The trend to undertaking true Impact Evaluations is sound, and utilizing the subsequent results will lead to greater accountability. The concept, however, continues to emerge and requires further education of the sector and implementation of “true” Impact Evaluations, as defined by International Initiative for Impact Evaluation (3ie; New Delhi, India),68 and for these reasons, Impact Evaluation remains in the comprehensive framework unchanged. The “diamonds,” as illustrated in Figures 1 and 2 in the Impact Evaluation section of the framework, are interpreted as indicative of Impact Evaluations, which could occur during any phase of the disaster timeline.
Figure 2. Comprehensive Framework for Disaster Evaluation Typologies 2018.
Abbreviations: GFDRR, Global Facility for Disaster Reduction and Recovery; GIS, Geographic Information Systems; H-EDRM, Health Emergency and Disaster Risk Management Framework; HIA, Health Impact Assessment; PDNA, Post-Disaster Needs Assessment; WHO, World Health Organization.
Accountability
Quantitative results on Accountability met only one of the four consensus thresholds (Table 1). Qualitative responses included 17 comments that provided constructive critique, which led to modifications in the comprehensive framework. Accountability was the most controversial element within CFDET 2017. Respondents’ qualitative comments demonstrated varied and different points of view, which reflects current debates surrounding Accountability. Reference Tan and von Schreeb69 Disagreement regarding definitions, terminology, and whether Accountability could be included as a stand-alone “evaluation typology” was supported by the responses received. Examples of these comments included: “There is no portrayal of accountability typologies at all, unless you are referring to the ‘up,’ ‘down,’ ‘across,’ ‘down’ typology of donors, stakeholders, and beneficiaries;” and “It would be worth linking accountability to both monitoring and evaluation typologies; ie, you do them for both learning as well as accountability.” These responses had the biggest impact on CFDET 2017.
Accountability and learning are often mentioned as two main goals of evaluation. Reference Cosgrave and Buchanan-Smith70 Accountability was included in CFDET 2017 because of its significance as a cross-cutting theme and the over-arching concept of effecting better quality and performance in humanitarian action. Since the 1990 s, a push to increase quality and Accountability exists in the humanitarian sector, including all aspects of work/interventions delivered to the communities and people affected 71–Reference Blagescu, de Las Casas and Lloyd73 by disasters. There is a moral imperative to ensure Accountability is at the forefront of all that is delivered in the disaster setting, whether that be to the affected community, donors, and stakeholders, however so defined.
Accountability as a stand-alone element was removed from CFDET 2017; however, the concept was included as an Outcome Typology in the updated comprehensive framework.
Evaluation Standards, Evidence, and Knowledge Management
Quantitative results on Evaluation Standards, Evidence, and Knowledge Management met two of the four consensus thresholds (Table 1). Qualitative responses were varied and included: “I think this is the ideal and what many organizations are striving towards, but I don’t believe it’s been widely adapted;” and “Ethical consideration should be placed in this domain.”
These results may reflect the current debates related to evaluation and to the way Evaluation Standards, Evidence, and Knowledge Management were illustrated within CFDET 2017. Importantly, CFDET 2017 illustrates contemporary disaster evaluation typologies rather than research methods, which classifies its own typologies. Ethical evaluation practice has been retained as a highlighted component of this element.
Minor modifications were made to this element to promote clarity and understanding. The first modification related to the position of the three individual boxes highlighting the independent concepts of Evaluation Standards and Guidelines; Evidence-Based Reviews and Registries; and Knowledge Management. These three boxes have been moved and included within a single, larger box to indicate their relationships to one another and that these concepts (under the umbrella of Evaluation Standards, Evidence, and Knowledge Management) are applicable across the entire framework.
The second and third modifications are related to comments regarding “Knowledge Management.” “Cross-sectoral research” has been replaced with “cross-sectoral collaboration,” and the term “translation” added. A major aim of the updated comprehensive framework encourages cross-sectoral collaboration on Knowledge Management and, importantly, promotes the translation of evaluation findings into improved policy and practice, which may lead to improved outcomes for disaster-impacted communities. Reference Turoff74,Reference Raman, Kuppusamy, Dorasamy and Nair75
Within this element, four new evaluation standards/guidelines were identified in addition to those previously published in PDM; Reference Wong, Spencer, Boyd, Burkle and Archer1 namely:
Moore, et al: Process Evaluation of Complex Interventions: Medical Research Council Guidance; Reference Moore, Audrey and Barker76
Christoplos, et al: Strengthening the Quality of Evidence in Humanitarian Evaluations. ALNAP Method Note; Reference Christoplos, Knox Clarke, Cosgrave, Bonino and Alexander77
Christoplos I, Dillon N, Bonino F: Evaluation of Protection in Humanitarian Action. ALNAP Method Note; Reference Christoplos, Dillon and Bonino78 and
Blanchet, et al: Using Research Evidence in the Humanitarian Sector: A Practice Guide. Reference Blanchet, Allen and Breckon79
Targets and Indicators as a Cross-Cutting Theme
As previously mentioned, targets and indicators, and their measurement, represent an emerging maturity in the sector. A Baseline Typology, “preparedness surveys,” included in CFDET 2017, now demonstrates more sophisticated, updated exemplars: national surveys in Canada,Reference Taylor-Butts80 the USA, 81 and a state survey in Queensland, Australia. 82 This typology was included in CFDET 2017 and no changes were made, other than to identify these recent exemplars in the text of the discussion and adding them to the updated bibliography.
“Strengthening Resilience” in the CFDET 2017 Core Structure included the Rockefeller 100RC Framework. Evolving from this international framework has been the “City Resilience Index” as a companion typology 83 designed to measure resilience within the Rockefeller City Resilience setting and was added to the framework. Although influential in guiding the Rockefeller 100 Resilient Cities project, it may be too early to determine its use or its influence on resilience in those cities. In Australia, an Australian Natural Disaster Resilience Index is also evolving; but again, it is too early to determine its use or its influence on resilience in Australia. Reference Parsons84–Reference Parsons, Morley and McGregor86
The “Recovery Indicators” typology emerged recently for use in the recovery phase of the disaster timelines and has been added to the Outcome Typologies. The recent literature suggests an attempt to identify factors that may lead to effective or good recovery and strategies to monitor the communities progress towards recovery after major events. This approach evolved into developing validated indicators of disaster recovery. In the USA, Horney led an extensive process to validate key indicators for disaster recovery. Reference Horney, Dwyer, Chirra, McCarthy, Shafer and Smith87 In the Australian setting, a set of outcomes, indicators, and standards of successful recovery are referenced from the Australian Monitoring and Evaluation Framework for Disaster Recovery Programs. Reference Argyrous34 These shape the future direction of data collection and measurement in this sector.
Comprehensive Framework for Disaster Evaluation Typologies 2018 (CFDET 2018)
The outcomes of the validation process of CFDET 2017 provided opportunities to improve the framework through its readability, understanding, and usability. Modifications were made based on feedback obtained, and the resulting comprehensive framework is shown in Figure 2. Strong acceptance and support for CFDET 2017 as a unifying framework demonstrated it cohesively represented the relationships between the different evaluation typologies.
Limitations
Important lessons were learned while implementing the online survey, which will be valuable for future surveys. Some potential limitations of this validation survey included: the difficulty in attracting an acceptable number of participants, which became very time consuming and prolonged the duration of the survey; the electronic format of the survey and the quality of the technology that was used by different participants which restricted their responses; the estimated time to complete the survey (30-45 minutes); and the fact that the accompanying paper Disaster Metrics: A Comprehensive Framework for Disaster Evaluation Typologies describing the development of CFDET 2017 Reference Wong, Spencer, Boyd, Burkle and Archer1 was “in press” and not actually published at the time of the survey. The potential for self-selection bias exists, given the high non-response rate. Despite this, saturation of data and meeting the consensus thresholds of most elements in the framework were evident in the contributor responses, which is testament to the validity of the results. The number of responders is consistent with acceptable levels for similar research studies.
Future Actions
Given that the comprehensive framework has now been validated and improved, the direction for future actions include:
Preparing an interpretive guide to accompany CFDET 2018;
The implementation of toolkits to support disaster evaluation typologies; for example, “How to Guides,” of which the WADEM Research and Evaluation Frameworks Reference Birnbaum, Daily, O’Rourke and Loretti10–Reference Birnbaum, Daily, O’Rourke and Loretti18 and Evidence Aid’s (Oxford, UK) Use of Evidence in the Humanitarian Sector: A Practice Guide Reference Blanchet, Allen and Breckon79 are leading examples;
Collaboration with international organizations, such as WADEM and Evidence Aid, to schedule promotion, training, and implementation of CFDET 2018;
Facilitate, support, and publish an initial suite of disaster evaluations using the comprehensive framework to structure the evaluation methodology;
Evaluating the link between CFDET 2018 and the significant international disaster risk reduction frameworks identified in the Core Structure; and
Monitoring and evaluating the use and usefulness of CFDET 2018.
Conclusion
The objective of the international peer-reviewed validation process of CFDET 2017 was to seek acceptance and feedback from experts in the fields of disasters and emergencies; humanitarian and development; and evaluation. This objective has been achieved. For the first time, a framework of this nature in the disaster setting has undergone a rigorous validation process at the international level and is supported by experts in three related disciplines. Respondent feedback obtained during the validation process improved the comprehensive framework. Modifications have enhanced the framework and added strength to the framework’s elements, understanding, and usability. The updated CFDET 2018 provides a unique, unifying framework within which existing typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting. Additionally, it facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. It is expected that CFDET 2018 will improve policy and practice associated with the delivery of health interventions in the disaster setting by strengthening the evidence-base for interventions delivered across the disaster spectrum. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.
Acknowledgements
The validation process would not have been possible without the input from all those who responded to the survey. This contribution is acknowledged and appreciated, and it will help advance disaster evaluations. A special thanks to Jackie Van Dam and Dr. Sarah Wong for their assistance.
Conflicts of interest
none
Supplementary Material
To view supplementary material for this article, please visit https://dx.doi.org/10.1017/S1049023X19004400
