Theme 1: Experiential Learning – Incorporating Realism into Training and Exercises

Lessons learned from real-world events and exercises have frequently served as the premise for establishing training priorities. While traditional instructional design entails passive learning, training design has evolved to incorporate active learning formats that not only provide students with opportunities to apply individual competencies for translating KSAA into actions, but also reflect on the outcomes of their actions to identify training needs and priorities.

Experiential learning serves as a prominent example of active learning for public health emergency response training. It allows students – or leaders in the context of this review – not only to apply leadership competencies to make incident management decisions or react/adapt to fluid emergency conditions, but also to evaluate the effectiveness of their decisions in minimizing adverse outcomes of an incident within context (eg, austere environments). In other words, the design and application of experiential learning for developing incident management leaders can be particularly impactful given the low-probability but high-consequence nature of public health emergencies and disasters. ^{Reference Silenas, Akins and Parrish603}

In the context of experiential learning to facilitate immersive and context-specific learning environments, educational games and simulation enhance training and exercise design and conduct as well as the assessment of technical competencies and personnel readiness in disaster medicine. ^{Reference Araz, Jehn and Lant19,Reference Olson, Hoeppner and Scaletta249–Reference Olson, Scheller and Wey251,Reference Barnett, Everly and Parker289} For example, the Uniform Services University of the Health Sciences developed a military medicine training curriculum that consists of low-, mid-, and high-fidelity simulations as experiential learning tools to emulate combat field conditions. ^{Reference Goolsby and Deering217} To standardize disaster medical assistance team training, the Australian Medical Assistance Team training incorporated extensive immersion training and exercise activities to assess personnel readiness. ^{Reference Norton648} In addition, modeling and simulation tools augmented VR training for Ebola virus disease management in health care settings, immersive simulations for pandemic influenza preparedness training, and incident management training for decision makers. ^{Reference Rega and Fink131,Reference Tena-Chollet, Tixier and Dandrieux159,Reference Sanjay259,Reference Ragazzoni, Ingrassia and Echeverri349}

In application, the effectiveness of experiential learning relies upon the design (focusing on the learning environment) and delivery (focusing on the learner) of the educational intervention (ie, training and exercises). Specifically, employing PBL as a learning theory in training design provides a structured framework that facilitates data-driven and context-specific (eg, case-based scenario) analytical thinking to achieve learning objectives. With continuing advancements in and growing adaptation of technological innovations for training delivery, modeling and simulation (eg, disease propagation models, VR) add realism, context, and the perspective of austere environments to enhance the learner experience. ^{Reference Stalker, Cullen and Kloesel155,Reference Goolsby and Deering217,Reference Streichert, O’Carroll and Gordon362} In conjunction, application of structured learning theories and technological tools extends experiential learning as a meaningful practice for reinforcing KSAA among training and exercise participants. For example, various disciplines have used PBL to provide emergency manager training for decision-making under stressful conditions, ranging from managing severe weather events to responding to bioterrorist attacks. ^{Reference Stalker, Cullen and Kloesel155,Reference Streichert, O’Carroll and Gordon362}

While there is tremendous value in experiential learning, this discussion does not discount the merit of passive learning through traditional instructional design (eg, classroom-based learning). While it does not entail an active participatory role among students, passive learning has established value in communicating new and vast quantities of information to large audiences. In the context of developing public health emergency response leaders, this discussion also recognizes the combined benefits of experiential learning (active learning) and traditional instructional design (passive learning) as an extension of blended learning to aid individuals in translating leadership competencies gained in a classroom setting to practical actions through context-specific or simulated environments. For example, in modernizing the military’s pedagogical methods for specialized unit training, the US Army adopts blended learning to integrate traditional instructional design and technological solutions to provide context-specific training. ^{Reference Plifka395}

Theme 2: Technological Adaptation – Describing the Evolution of Human-Driven Leadership to Technology-Assisted Leadership

Fluid crisis conditions require flexible and informed decision-making among emergency response leaders. The advent of and perpetual advancements in technological innovations have transformed crisis decision-making in recent years. While the emergency management community has increasingly embraced technological tools (see Figure 2), the capabilities of such tools have also expanded. For example, computer- or web-based tools that originated with information capture, presentation, and dissemination for maintaining situational awareness (eg, WebEOC, geographic information systems [GIS]) have evolved into complex decision support systems that encompass modeling and simulation capabilities to help decision makers plan for, respond to, and recover from disasters. Existing literature described decision support systems as a broad spectrum of technological innovation and tools, ranging from spreadsheet-based tools to commercially available software platforms. In the context of emergency response and crisis management, the recent transformation of crisis decision-making through the expansion and advancements in technology-aided decision support tools encompasses data integration platforms as well as sophisticated modeling and simulation capabilities to inform incident management decisions. For example, a seismic damage estimation case study in Canada integrates GIS and a specialized seismic risk assessment tool to provide the basis for identifying vulnerable areas and supporting risk management decisions. ^{Reference Alam, Tesfamariam and Alam285} A diverse range of smartphone applications from public and private sector entities provides emergency notification, public information, and data visualization capabilities to support disaster and emergency response. ^{Reference Bachmann, Jamison and Martin409} In addition, a policy informatics system using a simulation-based model integrates information from multiple data sources to support decision-making at the strategic and policy levels. ^{Reference Barrett, Eubank and Marathe410}

The growing adaptation of technology-based decision support tools also includes simulation-based resource modeling for public health preparedness and response, logistics support in humanitarian assistance, GIS-based spatial decision support, situational awareness for incident managers, and data mining and aggregation through social media platforms. ^{Reference Araz and Jehn18,Reference Peng, Zhang and Tang127,Reference Sharma147,Reference Taniguchi, Ferreira and Nicholson158,Reference Leaming, Adoff and Terndrup227,Reference Little, Loggins and Wallace235,Reference Stein, Rudge and Coker264,Reference Chalfant and Comfort418,Reference Hupert, Mushlin and Callahan434,Reference Chang and Li507,Reference McCormick522,Reference Ren, Wu and Hao600}

Reflecting on the growing trend in technological adaptation that may be attributed to the emergence of technological innovations and increasing output in literature over time, the evolution of human-driven to technology-assisted leadership has considerable implications on (1) research and development of technological tools for crisis management, (2) training on the proper and practical application of technological tools in crisis management, and (3) technology-aided design and delivery of educational interventions (ie, training and exercises), in general.

Decision support systems can inform decision-making in crisis management. As decision makers react and adapt to changing crisis conditions, decision support systems can provide these leaders with modeling or real-time data to inform incident management decisions. In addition, given the technical capability to simulate incident management decisions, decision support systems can allow decision makers to evaluate the outcome of those decisions before implementing them. In this respect, the role of decision makers or leaders is critical in the research and development (eg, assess needs, identify capability requirements, develop/improve user-interface), testing, revision, validation, acceptance, and deployment of new or existing decision support systems. ^{Reference Jennings and Arlikatti223,Reference LeMay699}

The growing prevalence of decision support systems, including through modeling and simulation support, is a prominent example of the technology-aided leadership approach. In parallel, technology-aided training design and delivery play an important role for developing public health emergency response leaders. Evident in training and exercises across multiple disciplines, technology-aided training design and delivery have a proven record in distance learning, self-paced learning, and interactive learning through web-based or mobile platforms, as well as in audio or video formats, particularly in reaching larger audiences. Technology-aided training programs are not only appropriate for customary workforce development programs, but also valuable for just-in-time-training during emergency responses. Notably, this review showed that the incorporation of technology in training and exercise activities in the context of developing public health leaders is promising but comprised only a relatively small portion of articles focused on the impact of technology. As such, the evolution of human-driven leadership to technology-assisted leadership centers on the increasing trend and ubiquity of technological tools to support decision-making. In other words, while evidence of the positive impact of technology is emerging, the preponderance of literature does not provide definitive and generalizable evidence.

Recognizing the value of technology-based tools and technology-aided training programs, the authors also acknowledge the potential limitations of technological innovations. Decision makers or leaders should not rely solely on their modeling and simulation capabilities to inform incident management decisions. Rather, decision makers or leaders should concede the unpredictability of real-world incidents and react or adapt incident management decisions to fluid emergency conditions.

In the context of this study for developing public health response leaders in incident management through educational interventions, achieving the true potential of technological innovations and adaptation requires the integration of pedagogy and technology.

Theme 3: Educational Continuum – Incorporating Leadership Training in Public Health Academic and Professional Development Curricula

The educational continuum represents the continuous process of learning, which starts with degree-granting programs and professional schools and proceeds through continuing, on-the-job training. Currently, most leadership training leans toward postgraduate or job-related training. ^{Reference Mackinnon, Bacon and Cortellessa116,Reference Hall, Ward and Cunningham309,Reference Shield, Wiesner and Curran490,Reference Horney and Wilfert662,Reference Scutchfield, Spain and Pointer678} Examples of job-related leadership training include the Public Health Leadership Institute, a program designed to expand and enhance the leadership skills of senior public health officials, ^{Reference Scutchfield, Spain and Pointer678} and a distance learning series of current preparedness topics delivered via a synchronous web-conferencing system by the Northwest Center for Public Health Practice at the University of Washington. In contrast, fewer examples of leadership training take place earlier in the educational continuum during formal schooling. ^{Reference Orfaly, Biddinger and Burstein338,Reference Veenema681}

The process by which future leaders gain competencies must be developed and reinforced throughout the educational continuum in a systematic manner. Currently, training is often developed in response to urgent needs rather than derived from established competencies. ^{Reference Von Lubitz, Carrasco and Fausone269,Reference Li, Li and Chen324} Integration of leadership training throughout the educational continuum would lead to increased standardization, ensuring that future leaders acquire leadership skills systematically. Targeted leadership training should also be incorporated at all levels. Completing training at leadership institutes (eg, National Public Health Leadership Institute) promotes collaborative leadership and building knowledge-sharing and problem-solving networks. ^{Reference Umble, Steffen and Porter669}

The literature also highlights the value of multidisciplinary training. ^{Reference Adini, Goldberg and Cohen6,Reference Hoying, Farra and Mainous485,Reference Farra, Smith and Bashaw622,Reference Neudorf, Obayan and Anderson664,Reference Potter, Burns and Barron666,Reference van der Haar, Koeslag-Kreunen and Euwe670} As public health emergency response requires extensive cooperation and coordination, often involving multiple disciplines and diverging perspectives, multidisciplinary training can help identify coordination challenges and potential solutions that a single discipline or area of focus may miss or narrowly address. As such, understanding the interdependencies of multidisciplinary decision-making is critical to the success of public health emergency response leaders. ^{Reference van der Haar, Koeslag-Kreunen and Euwe670}

Collaborative partnerships among governmental agencies, academia, and industry continue to emerge. These affiliations are a driving force in developing future training to meet evolving leadership needs (eg, develop a better understanding of the role of technology in leadership decision support and decision-making). For example, CDC-sponsored PERLCs focused on the development, delivery, and evaluation of targeted learning programs for enhancing workforce readiness.

Those who serve in leadership positions bring considerably different levels of prior training and experience to the task. Newly designated leaders in the public health emergency response context may lack critical KSAA to perform effectively. Similarly, these individuals may lack the relevant KSAA to manage numerous individuals and large organizations effectively during a response.

While leadership training is an essential element of public health workforce development, systematic efforts to interweave leadership training specifically geared for public health emergency response vis-a-vis traditional training or educational curricula have not widely occurred. Public health leaders put to the test during crises have shown that subject matter expertise alone does not necessarily correlate with effective leadership. Accordingly, public health practitioners who may fulfill leadership roles or positions in public health emergency response should engage in leadership education and training throughout their entire educational continuum, starting with public health academic degree programs.

Establishing a blueprint (eg, beginning with foundational disaster preparedness and management concepts and culminating in advanced methods to manage complex and multidisciplinary responses) for developing future leaders in public health emergency response increases uniformity of training practices and consistent translation of KSAA into decision-making practices. Such an educational framework can aim at (1) building and refining comprehensive curricula suited for leaders from the earliest stages of career development, (2) incorporating the curricula into formal academic programs, and (3) integrating the curricula with specific job-related training for public health emergency response leaders.

Theme 4: Defining Leadership Holistically – Developing Leaders through Competency and Character

Technical competencies among responders are an implicit requirement of public health emergency response. At the same time, the fluidity and/or austerity of emergency conditions often demand leaders to work outside of their comfort zone, thus requiring adaptability and flexibility to make incident management decisions that correspond to the present operational environment. ^{Reference Mantha, Coggins and Mahadevan386}

Traditionally, technical competencies achieved primarily through specialized education and levels of experiences have encompassed leadership. Recognizing this established approach, the authors also believed that existing literature presented an emerging perspective on leadership traits that places equal emphasis on the character or soft skills of an individual. ^{Reference Cools and Van Den Broeck47,Reference Cooper49,Reference Crichton and Flin50,Reference Smith and Kuldau359,Reference Curnow, Mulvaney and Calderon738}

Increasingly prevalent in military strategies and emerging leadership studies, existing literature suggests that character encompasses personality (eg, resiliency under stress), cognitive abilities (eg, analytical thinking, problem solving, decision-making), and interpersonal skills (eg, communication, receptiveness of diverging views). ^{Reference Cools and Van Den Broeck47,Reference Masotti and Appleget382,Reference Rodríguez Montequín, Mesa Fernández and Balsera399,Reference Tremble and Trueman608,Reference Mueller-Hanson, White and Dorsey646,Reference Cojocar736} In the broad context of crisis management and emergency response, these traits comprise the emotional intelligence of leaders and enable them to apply technical competencies and adapt to external environments. ^{Reference Mackinnon, Bacon and Cortellessa116,Reference Shuck and Herd570}

Recognition of competency and character as the defining traits of leaders stands in contrast to the traditional emphasis on technical competencies for leaders. Application of established educational models or approaches is valuable for developing the competency and character of public health emergency response leaders in incident management.

As discussed in Theme 1, experiential learning, particularly with the aid of modeling and simulation tools, may also allow students to develop character collaboratively by applying personal qualities and exercising cognitive and interpersonal skills, as well as identifying potential individual deficiencies among these traits.

In addition, to serve as public health emergency response leaders in incident management, position descriptions specify the required technical or domain-specific competencies (eg, epidemiology, infectious disease). However, there is limited evidence to support standardized instruments specific to measuring leadership character traits in public health emergency response.

This discussion extends to the challenge in selecting leadership candidates, specifically based on the need for specific character traits. In the context of military leadership studies, existing literature discusses the potential utility of structured psychometric assessments (eg, Myers-Briggs Type Indicator® [MBTI®]). ^{Reference Masotti and Appleget382,Reference Silenas, Akins and Parrish603,Reference Pisapia and Coukos-Semmel687,Reference Seiler and Andreas688} Although there is value in such tools for recognizing personality types, this review infers that the suitability (eg, costs, potential bias for or against certain personality types) of using such assessments as indicators of leadership character traits, predictors of performance, and selection criteria for leaders warrants further examination. For public health emergency response, a framework for character traits among incident management leaders may be considered. Specifically, this leadership character framework may describe the relevant traits as a career maturity process through which individuals can develop these traits from a combination of domain-specific education tailored to and experiences from serving as incident management leaders.

Theme 5: Evaluation and Measures of Effectiveness in Training and Exercises – Reflecting on Current Practices and Opportunities for Progress

A critical finding from this scoping review is the relative weaknesses in evaluation and measures of effectiveness of training and exercises. Although many articles broadly touched upon the topics of evaluation and effectiveness, relatively few incorporated actual evaluations; even fewer still incorporated formal measures of effectiveness. ^{Reference Ablah, Hawley and Konda500} Except for a few notable articles, there was limited discussion of adaptable frameworks, specific instruments, or processes that may be applicable to consistently evaluate and measure effectiveness of training public health emergency response leaders. In particular, a glaring absence was the lack of discussion on cost-effectiveness, as only a handful of studies referenced distance technology, computer simulation, and gaming technology as potentially cost-effective training options. ^{Reference Ma, Mao and Zhou237,Reference Williams-Bell, Murphy and Kapralos274,Reference Demers, Mamary and Ebin480}

In addition, the majority of literature from this review included mostly descriptive and qualitative studies, narratives, and informational articles; and the predominant form of evaluation included self-reported gains and pre- and post-intervention surveys. ^{Reference Rafferty-Semon, Jarzembak and Shanholtzer256,Reference Daniels, Farquhar and Nathanson298,Reference McCabe, Everly and Brown592,Reference Abatemarco, Beckley and Borjan612,Reference Ablah, Molgaard and Fredrickson613,Reference Farra, Smith and Bashaw622,Reference Qureshi, Gershon and Merrill630,Reference Dausey and Moore750,Reference Rask, Gitomer and Spell768} There were relatively few quantitative or data-driven studies. ^{Reference Chen and Pena-Mora37,Reference Chiu, Polivka and Stanley479} Moreover, very few articles reported directly on real-world responses as many of the informational articles on this topic, by nature, did not involve study design but were included by the reviewers as a reflection of the emerging nature of this key theme. While not to detract from the clear value of this body of work, it does demonstrate the tremendous degree of difficulty in performing, documenting, and disseminating the results of meaningful evaluation, especially those taken from real-world events, in a manner that can objectively inform future training and exercises.

The heterogeneity of training and exercise designs further limits the ability to generalize conclusions regarding effectiveness. In other words, when training or exercises activities are not developed systematically or structured from well-established competencies with clearly defined learning objectives and success criteria, efforts at evaluation prove difficult. Without formal evaluation processes, assessing the effectiveness of training and exercises is challenging at best and limited in scope. While qualitative evaluation may inform the prospective design of training or exercises, these assessments are highly subjective in nature, especially in the absence of metrics. For example, discrepancies in the implementation of the current exercise evaluation process (ie, Homeland Security Exercise and Evaluation Program) call for consistent and systematic evaluation metrics to assess leadership performance in this context.

The sparse number of studies involving quantitative analysis of real-world response highlights the complexities of performing meaningful evaluation or developing measures of effectiveness. To address this challenge, automated information collection and pre-positioned methods for data capture coupled with dedicated resources for post-event analysis may yield critical insights for future responses and training. In addition, comparative studies of cost-benefit analyses for various types of training and exercises may also be an important area for further exploration.

Strengths and Limitations

This scoping review describes the landscape of educational interventions that applies to developing public health emergency response leaders in incident management. Strengths of this review include analyses of relevant articles not only in public health practice, but also in additional disciplines (see Methods) to glean applicable content that may benefit the development of public health emergency response leaders through educational interventions. To recognize trends in relevant professional development opportunities, this review considered a broad range in publication dates between January 1990 and October 2017.

Reflecting on the broad time parameter and diverse disciplines that comprised this review, 1 limitation relates to existing literature encompassing the broad and liberal use of the term best practice to describe educational practices that may be successful, effective, preferred, or promising. As such, the precise meaning and objective measures to classify educational best practice warrant further exploration. In addition, relevant articles published since the original time-parameter used in this review were not considered. Finally, articles included in this review were focused primarily (85%) on the United States, so insights gathered from this review may not apply fully in international contexts, particularly in nations where public health preparedness and response systems and capabilities are not structured comparably as those in the United States.