Recent historical events have changed the global security landscape. The US government has refocused efforts toward stabilizing fragile, failing, or postconflict states around the world. Since the end of the Cold War, the US Department of Defense (DoD) has increasingly conducted military-led missions designed to improve stabilization especially in countries in which essential (water, health, food, shelter, sanitation) and social services have been disrupted.1 As such, the DoD currently and actively incorporates health sector projects in these military operations.
In 2004 the Bush administration, in an effort to ensure a broad US security agenda after the terrorist events of September 11, 2001, created the Office of the Coordinator for Reconstruction and Stabilization to identify, consolidate, and direct available US resources needed to meet emerging global challenges and promote long-term stability in at-risk countries.2, 3 In November 2005 a DoD directive defined “stability operations” as a “core US military mission” with a “priority comparable to combat operations.”4 This necessitated an expanding role for the DoD from traditional military strategy to one that includes stability operations in all phases of mission planning. Stability operations, broadly defined, are designed to prevent or mitigate both preconflict and postconflict collapse of failing states and include “security, transition, counterinsurgency, peacemaking, and the other operations needed to deal with modern day security challenges.”5
Inadequate health care and public health infrastructure in developing nations is a root cause of conflict and instability.6 The DoD has purposely incorporated health sector planning in stability operations to increase security capacity and prevent further slippage toward conflict or war.Reference MacMillan7 Health-related projects are collectively lumped into “humanitarian assistance” (HA) by the DoD. These projects are designed to assist the host nation (HN) with development and refinement of health programs and medical and public health infrastructure, basic necessities of life, and health and sanitation services, as well as monitoring health indicators and health risks.1 In coordination with other US agencies, HNs, and the international development community, the DoD is in a position to accelerate both HN basic services and governance.Reference MacMillan7, 8
Historically, DoD HA projects have been ad hoc, short-term, 1-time interventions, limited in their ability to show effectiveness.8, Reference Drifmeyer and Llewellyn9 Counter to emerging international humanitarian assistance standards, measures for impact assessment are rarely collected during DoD HA operations, and when attempted, the measures are often incomplete and limited to outputs or achievement measures.Reference Drifmeyer and Llewellyn9 The current DoD HA measures provide only “count data,” such as number of medical and surgical patients seen, number of immunizations given, quantity of pharmaceuticals prescribed, number of Meals Ready to Eat delivered, or number of health clinics built. This type of data lacks baseline measurements, an identifiable population denominator, and outcome goals for comparison of results. More important, these data do not indicate favorable or unfavorable rate changes on population health measures, such as mortality or morbidity. DoD HA efforts, although undertaken with good intentionions, may have secondary or tertiary effects that transcend the primary intervention and may duplicate or contradict activities of other agencies.6 As HA operations assume an ever greater role in US military strategy, the DoD must strengthen its comprehensive, results-oriented HA planning toward activities that emphasize partnerships and cooperation with the HN, their citizens, and the national and international development community.3,8–Reference Drifmeyer and Llewellyn10
As an initial step in developing and implementing evidence-based HA programs in military-led missions with the HNs, methodologies used by both the international aid community and DoD resources were analyzed using qualitative gap analysis and impact assessment studies.
METHODS
A multistep approach was used to identify and compare methodologies.
International Aid Community Evidence-based Literature Review
Aid community standards were selected because of their field-level validation and reliability in interagency comparison studies. Furthermore, adoption of these standards by the DoD would be necessary if transition of programs to the aid community or HN were to occur. A literature search of peer-reviewed reports was conducted to ascertain current impact assessment methods used by the international aid community with findings formatted as a template for further comparison.Reference Brennan and Nandy11–Reference Beitler, Junnila and Meyer24
DoD After-action Reports and Lessons Learned
To investigate how the DoD collects, analyzes, and reports data and information on HA operations, an information search was conducted using a Web-based database for US-only DoD service-specific medical after-action reports (AARs) and lessons learned (LLs) on DoD HA operations.25–29 Also, DoD HA subject matter expert (SME) interviews and working group discussions were conducted. To compare the process of impact assessment between the DoD and validated civilian aid community standards, information obtained during each search was summarized and qualitatively organized into tables.
To provide lessons from an HA operation, AARs are routinely generated by all service-specific military units following deployment and recorded as formal service-specific medical LLs. These reports were selected as reliable search resources because they are the only known and accessible repository of data and information on DoD HA operations. Each report provides background mission information, medical services role, and LLs.
LLs are written with a standardized format including a topic, observation, discussion, and recommendation. Searchable medical function areas for LLs include command and control, communication, administration, intelligence, force health protection, logistics, ancillary clinical services, medical regulating, and patient care. The LLs Web-based organizational format includes identification number, date, lesson learned topic or title, command name and unit, and military campaign. All Web links are available at the Center for Army Lessons Learned, but are not in the public domain.30
Ten structured, search engine Web searches of each service-specific database were performed using the search terms “humanitarian assistance” (HA), “humanitarian assistance/disaster relief” (HA/DR), “medical civilian aid/assistance projects” (MEDCAP), “dental civilian aid/assistance projects” (DENCAP), “medical readiness training exercise” (MEDRETE), “medical civilian-military operation” (MCMO), “military operations other than war” (MOOTW), “patient care,” “vaccine,” “impact assessment,” and “measures of effectiveness” (MOEs). The search terms “MEDCAP,” “DENCAP,” “MEDRETE,” “MCMO,” and “MOOTW” are military operations specific to DoD HA. The terms “patient care” and “vaccine” were used to broaden search results and link military operations to health sector humanitarian assistance.
AARs and LLs were reviewed to identify key terms related to HA activities and impact assessment. Medical AARs were first reviewed by subject title, the mission description, the date, military operation, and unit name. All of the reports were reviewed for population health indicators, quantifiable health sector humanitarian assistance activity output, and outcome data (including patient care, medical or surgical cases, public health, and percent of change), measures of effectiveness or impact assessment metrics, and links to larger security and strategic goals.31 Reports also were reviewed for descriptive characteristics, including mission duration and type of medical treatment specialties provided.
DoD HA Subject Matter Expert Interviews
This study considered interviewees as DoD SMEs if they participated in 1 or more DoD HA operations as medical service planners, health providers, or research analysts. Twelve SMEs from Naval Health Research Center, Center for Naval Analysis, Bureau of Medicine and Surgery, Navy Environmental Preventive Medicine Unit-6, III Marine Expeditionary Force, Center for Disaster and Humanitarian Assistance Medicine, and Office of the Assistant Secretary of Defense for Health Affairs were identified and interviewed in 2006 and 2007. Interview questions and SME responses were qualitatively organized into thematic topic areas specific to DoD HA operations that include planning, data collection methods, data analysis, and impact assessment and reporting.
RESULTS
A review of 1000 reports that combined medical AARs and corresponding LLs from 1996 to 2007 identified a total of 178 reports using the key search terms: 38 (3.8%) AARs and 140 (14%) LLs. Only 7 (0.7%) reports refer to, but do not discuss, impact assessment or measures of effectiveness. Table 1 lists only the AARs identified per search term by number and percent of total reports reviewed. The remaining search terms did not yield AARs.
TABLE 1 Summary of After-action Reports Identified by Key Search Terms
The search term “HA/DR” also identified 140 (14%) separate LLs. Seven (0.7%) AARs included key terminology in assessing the impact of DoD HA missions. Three (0.3%) were MEDCAP AARs and 4 (0.4%) were HA/DR AARs. Of the 7, the use of the term impact assessment was mentioned 1 time and the term measures of effectiveness 3 times. In the AAR discussing impact of HA activities, the definition of impact was not provided. Of the AARs discussing measures of effectiveness, 1 recommended the need for their use, 1 was in reference to Iraq security reconstruction not health sector HA, and the other provided a blank spreadsheet template for recording measures of effectiveness. The remaining AARs were narratives of medical activities in context of military operations other than HA.
Table 2 summarizes routinely documented medical AAR and LL statements and the information necessary to bridge the gap in DoD HA impact assessment to meet international standards. AAR summary statements (first column) are compared with civilian standards (second column). Table 2 summary statements are divided into a timeline to conceptualize HA into planning (before), execution (during), and analysis (after). This timeline follows normal military mission strategy and is parallel to impact assessment models used by international development agencies.32–35 It represents the most critical steps within monitoring and evaluation systems36 and logic37 models used by the US Agency for International Development, Office of Foreign Disaster Assistance, and Department of Health and Human Services, as well as the international business community, the World Bank, and numerous nongovernmental and international organizations in HA programs.
TABLE 2 Summary of After-action Report Statements and Information Gap Linking to International Standards in Impact Assessment
The following italicized statements are listed verbatim from AARs and summarized in the first column of Table 2. The text following each statement discusses how these statements fall short of meeting international standards and/or expectations and are summarized in the second column of Table 2:
“Because of time and budget constraints, only a percentage of those cases were actually chosen for surgery.”
The surgical needs from all-type surgery patients from the total HN population were not identified nor was the actual percentage of patients treated. No reference was made to the target population (denominator data), no systematic method of identifying patients with surgical needs was established, nor is it described how the select few that were treated were identified and chosen to receive surgical services. The statement also highlights the short-term nature of DoD HA operations and the potential consequences of providing medical care during brief visits in recipient HNs.
“As a result of personnel shortfall, we were only able to run 3 out of 12 operating rooms a day.”
No patient population needs assessment was completed before the HA team planning or arriving at the HN. The statement suggests manning requirements were short because the team could not operate at full capability regardless of whether those services were necessary to treat the HN population. The HA team’s capability to provide care did not match the needs of the HN. The statement also suggests the HN and other aid organizations were not involved in the planning process.
“We implemented a few process improvement initiatives and things flowed more smoothly.”
There is no mention of what process improvements were used or the outcome of that initiative. No data to support the statement were collected or reported before or after the process improvement initiative.
“Patients received free emergency, corrective, preventive, and educational services. A total of 8075 patients were seen and an estimated 16,661 treatment services were provided. This includes 674 veterinary patients, 1010 optometry patients, 1310 dental patients, and 4430 medical patients. This humanitarian care has an estimated value of $760,000 USD. The total mission supply budget was $50,000.”
This statement, like most AAR statements, lists output data (achievement indicators) with no clear link to how those services or costs affected outcomes for the recipient nation or DoD and HN goals. It is unclear whether services provided were appropriate for the HN or within internationally acceptable treatment practice guidelines.
“The MEDCAP portion of (exercise X) was an outstanding success both in the terms of patient care treatment and in meeting the objectives of the Global War on Terror. Eight remote village locations in the heart of ‘Island X’ were reached and over 7000 patients received health care benefits as a direct result of this mission.”
This statement lists output data (achievement indicators) only. There is no information link to program goals or longer term “outcome” of the services provided. There is no clear link to how “success” was measured specifically to patient care indices or Global War on Terror (GWOT) objectives, nor a list of the benefits expected to have been received by the HN because of the interventions provided by the HA team, nor a list of HA activities that could be reciprocated in future HA operations if in fact those activities do benefit the HN and meet the objectives of the GWOT.
“For many people, their quality of life was improved and their perceptions were changed by this medical mission.”
No formal quality-of-life measurement or percent of change was identified or recorded. This statement is anecdotal, nonverifiable, and not linked to mission objectives or outcome goals. A perception change may have occurred for either the AAR author or the HN population, but that assumption can only be made using the AAR narrative that is neither quantifiable nor reproducible. Neither statement lists baseline or final measurement to support it.
Table 3 qualitatively summarizes DoD SME responses organized by question topic asked during interviews. The first column of Table 3 lists SME interview questions organized by topic into HA planning, data collection and analysis, and impact assessment. The second column of Table 3 lists thematic SME responses to each question topic. The statements highlight SME recommendations and DoD HA operation shortfalls. Interviewee statements mimic those observed in AARs.
TABLE 3 Thematic Summary Responses of Subject Matter Experts (SMEs) to DoD Humanitarian Assistance Operation Interview Questions
DISCUSSION
Results of the DoD service-specific database search and SME interviews show a major gap in the incorporation of impact assessment by the DoD in execution of its HA activities compared with the international aid community. The Web-based search results suggest the DoD lacks a standardized HA AAR format and inconsistently generates corresponding appropriate quantifiable data and outcome-focused documentation before, during, and after operations. Conventional medical AARs are narrative and authors provide subjective comment on topics important to their medical expertise or personal experience during HA operations.Reference Drifmeyer and Llewellyn9 They also confirm that DoD HA operations are short-term, brief missions that are rarely analyzed for effectiveness. In addition, SMEs suggest the DoD HA operations are conducted with capability seldom matched to targeted HN needs, poorly planned with appropriate stakeholders, and usually fail to meet international standards for providing humanitarian aid.
Individual Combatant Commands are given control of planning and executing HA operations. The DoD does not have a unified approach to HA operation planning, execution, and analysis evident upon review of AARs and discussion with SMEs. Given the nearly 200 humanitarian-type operations performed annually by the DoD, the expectation to find more AARs and LLs using simple Web-based humanitarian assistance search terms was not met.Reference Drifmeyer and Llewellyn9, 38 Possible explanations for limited available AAR data include the exclusion of classified data, which has important logistical and strategic significance for the DoD, and proprietary information from DoD-contracted research centers. At minimum, this information exclusion highlights the need for the DoD to strengthen data collection and reporting using a unified, transparent format that can be shared within the DoD and the international development community.Reference Drifmeyer and Llewellyn10
Interviewees believed that the strongest yield from DoD HA operations could be capacity building by partnering with HN professionals and ministries of health to improve health parameters. This suggests that activities such as train-the-trainer and capacity building outweigh individual patient encounters; however, specific examples of capacity building activities were not shared during the SME interviews. This also suggests that engaging the HN in planning HA is necessary and by itself an activity that builds national and local capacity and fosters goodwill and HN confidence. SMEs were chosen as a convenience sample based on expertise in DoD HA; however, the interviewees may overly represent the US Navy, which is a potential source of bias in this analysis.
The DoD commonly uses output measures, referred to as “count data” or “achievement indicators,” which are frequently listed on AARs. For example, output data collected in 2006 by the hospital ship USNS Mercy during HA operations in Southeast Asia was used to predict hospital ship manning requirements and HA capability.Reference Konoske and Negus39 This is invaluable information to identify an organization’s ability to perform and complete predescribed tasks and is a necessary initial step when analyzing the impact of HA operations; however, output measures should not be used exclusively as effective measures of operation impact. These data may lack baseline measurements, an identifiable denominator, and outcome goals for comparison of results. Without a clearly defined guideline describing how to bridge this information gap, the DoD will continue to lack the ability to assess the impact of HA, fail to recognize the work of those involved in the project by not validating their performance, lack credibility in defending the worth of the HA intervention on the HN population, and lack capacity to transition military-led programs to international partners that routinely use outcome indicators.
Another measure of HA impact recently used by the DoD is public opinion. The “Terror Free Tomorrow” report compared successive public opinion poll intervals collected in the world’s largest Muslim countries, Indonesia and Bangladesh, on attitudes toward America favoritism following the Southeast Asian 2004 tsunami disaster relief missions performed by the US Navy.40 The report concluded that operations performed by US hospital ships were effective HA platforms and received favorably among the HN populations. DoD SME interviews identified HN public opinion as the single most important measure of HA impact. The public opinion poll report conclusions were confounded by activities and modalities performed by other HA agencies and were biased by comparing HA efforts between “disaster response” and “preplanned operations.” Immediately following the tsunami, relief efforts were rapidly provided by combatant navy ships and other military forces in the region. The USNS Mercy did not arrive in disaster-striken areas until 41 days after the event. The hospital ship mission was incorrectly labeled “disaster response.” The qualitative measures of effectiveness reported may have falsely identified USNS Mercy as the reason for HN favoritism toward US assistance when the HN positive opinion may well have been generated by the initial activities by combatant ships and forces. Caution should be exercised in directing valuable resources for HA operations supported only by qualitative data and that which is confounded by other variables known to affect the outcome desired. In this case, the report does not identify which HA activities directly contributed to favorable public opinion. To truly assess the impact of HA, a link of activities to outcome goals must be clearly identified, measured, and reported.
Conclusions
Using a review of unclassified, Web-based military AARs and LLs, and interviews with DoD SMEs, this qualitative analysis shows a major gap in the incorporation of HA impact assessment by the DoD in execution of its HA activities compared with the international aid community. DoD HA has been ad hoc, 1-time, brief operations focused largely on program output as an achievement only, as opposed to linking interventions to outcome and consensual strategic goals within the broader humanitarian assistance community. The DoD misses opportunities for participation in validated coordination to best benefit the HN and international aid community. Without a common data repository and analysis framework, there is a loss of institutional memory and learning from one HA operation to the next.
Authors' Disclosures The authors report no conflicts of interest.
