Results of the US Department of Defense (DoD) service-specific gap analysis conducted in Part I of this article suggests that the DoD does not implement impact assessments in execution of its health-sector humanitarian assistance (HA) activities that can complement, be shared with, or be transitioned to development assistance activities of the international relief community or host nation (HN).Reference Reaves, Schor and Burkle1 This oversight fails to meet the 2005 DoD Directive 3000.05 and recent Guidance for Employment of the Force (classified document) mandate calling for the application of measures of effectiveness in stability operations.2 Historically, DoD HA operations have been ad hoc, 1-time visit, brief operations focused largely on program achievements only as opposed to linking humanitarian interventions to outcome and consensual strategic goals within the broader humanitarian assistance community.

Each DoD Combatant Command is given control of planning and executing HA operations in its respective area of geographic responsibility. Guidance provided by the National Military Strategy and Quadrennial Defense Review calls on the DoD to synchronize and coordinate military activities and operations with international partners and nongovernmental organizations.3, 4 However, the DoD lacks a unified approach to HA operation planning, execution, and analysis and inconsistently generates corresponding appropriate quantifiable data and outcome-focused documentation before, during, or after HA operations.Reference Reaves, Schor and Burkle1 Given the growing need for joint DoD and coordinated international HA efforts, a common, well-defined framework for planning, executing, and assessing impact of humanitarian aid is necessary. To ensure ascribed internationally accepted minimum HA standards are met and international law is upheld by the DoD, the authors recommend implementation of an impact assessment model in DoD HA operations.

The intended goal of an impact assessment model for DoD HA operations is to bridge previously identified information gaps from traditional military input-output management toward a greater emphasis on outcomes.Reference Reaves, Schor and Burkle1 This model would assist policy makers and decision makers, commanders, HA program managers, and HN stakeholders in the ability to link the meaning behind documenting and counting output (achievement) measures and HA activities to what is actually accomplished (outcomes).

This impact assessment model can be applied to the level of an individual HA project, the humanitarian aid sector of interest, and to the broader strategic emphasis of the DoD. It uses both quantitative and qualitative measures and deductive/inductive reasoning to analyze impact, monitor and manage performance, and serve as a postmission evaluation of effectiveness process.Reference Reaves, Schor and Burkle1, Reference Burkle and Hayden5

SUPPORTING EVIDENCE FOR DoD USE OF AN IMPACT ASSESSMENT MODEL

The expected benefits from using this impact assessment model in HA are listed in Table 1. These benefits are derived from the literature regarding measures of effectiveness, program monitoring and evaluation (M&E), and impact assessment of HA.Reference Burkle, McGrady and Newett6–11 The benefits also directly complement the goals outlined in the 2006 Quadrennial Defense Review for upgrading the DoD to 21st century standards and are emphasized with an asterisk in Table 1. Benefits are organized chronologically into before, during, and after HA operations using conventional military mission strategies complementary to impact assessment model timelines.

TABLE 1 Benefits of Using an Impact Assessment Model in Humanitarian Assistance Operations for DoD

As shown in Table 1 and emphasized in Part I of this article, an impact assessment model provides valuable information to commanders and other decision makers so they can make evidence-based policy decisions, appropriately allocate resources, shape training, enhance program efficiency and effectiveness, and facilitate interoperability and timely transfer of humanitarian activities to the international aid community. Such a model would also assist the DoD chairman and Joint Chiefs of Staff in the ability to identify desired capabilities; define interrelated, joint military objectives and operating concepts; and assess risk for humanitarian aid in stability operations.3 Therefore, using an impact assessment model would minimize unintended consequences from poorly planned and executed HA activities, mitigate intended stability, and help answer which HA activities are successful, which are not, and why.

An impact assessment model would ensure that the DoD and its operational units are learning as an organization while expediting real-time application of evidence-based lessons learned to future stability operations. Reference Drifmeyer and Llewellyn8,12,13 This increased efficiency will help prevent the practice of repeating activities performed during previous HA operations simply because they were documented in after-action reports or lessons learned. Likewise, it promotes appropriate operational accountability. Commanders typically assign “causality” and reward their staffs for program outputs without a logical and traceable link from input to outcome. Due to the complex and often chaotic environment in which HA activities take place, completing all HA activities and outputs does not ensure that the desired outcomes and goals for improved health or US government strategies will necessarily be achieved. However, the process of measuring output indicators will provide a better means of assessing and documenting the reasons why.

Conventional military mission end-state strategies and short-term duration interventions both impede HA program mid-mission modification and counter the longevity necessary in development and sustainability. Applying the methods of impact assessment will align DoD HA activities with the longer duration sustainability needs of the HN and development community. Additional benefits of using the impact assessment model are in its ability to assist program implementation, monitoring, and evaluation at all levels of command. It provides a framework supporting transparent analysis, communication, and coordination both horizontally and vertically within the DoD chain-of-command and outside the DoD with key stakeholders.Reference Burkle and Hayden5

THE IMPACT ASSESSMENT MODEL

The impact assessment model is adapted from logic models (logframe) and shares the assessment concepts and terminology of the M&E system traditionally used in the business community and increasingly by aid development organizations, including the US Agency for International Development, Office of Foreign Disaster Assistance, and Department of Health and Human Services, along with the World Bank and numerous nongovernmental and international organizations in HA programs.7,8,10,11,Reference Roche14–23Figure 1 is a logic model overview that visually simplifies the sequential evolution of HA programs by dividing interventions into implementation (inputs, activities, and outputs) and results (outcomes and impact). Though shown sequentially, each step of the impact assessment model must be initiated, continuously reviewed, and sustained throughout the HA operation from stability operation inception and planning to military end-point and beyond to the transfer of DoD HA activities to the HN and/or national and international aid community. The bidirectional arrows between model steps emphasize the important influence each step has on the other; however, no individual step is more important than the other, and each step must be conceptualized as to how it fits into the humanitarian context in its entirety.Reference Mock and Garfield24, 25

FIGURE 1 Impact assessment model. Adapted from references7, 8, and 17.

The key terms of the impact assessment model, defined below, have been introduced to the DoD. 16,26,27 The terms “outputs,” “outcomes,” “impact,” and “results” are often used interchangeably by the DoD; however, these terms represent distinct yet interrelated steps within the impact assessment model and are shared among other national and international development agencies.

1. 1. Inputs: the financial, human, material, technological, and information resources used for the development intervention

2. 2. Activities: actions taken or work performed through which inputs—such as funds, technical assistance, and other types of resources—are mobilized to produce specific outputs

3. 3. Outputs: the products, capital goods or services, or changes that occur from development interventions

4. 4. Outcome: the likely or achieved short-term and medium effects of an intervention’s outputs

5. 5. Impact: the positive and negative, primary and secondary, long-term effects produced by a development intervention, directly or indirectly, intended or unintended

It is crucial to emphasize that outputs are results that assist HA program managers in evaluating project implementation, reflecting the ability of an aid organization to execute and complete HA activities as planned. DoD HA teams are customarily adept at executing HA activities linking resource inputs with outputs. However, output measures, termed “measures of performance” by the DoD, do not tell program managers or policy decision makers how those outputs translate into outcomes. This model provides the concept necessary for the DoD to link HA activities with desired results and ultimately to long-term strategic impact.

IMPLEMENTATION OF THE IMPACT ASSESSMENT MODEL

A January 2008 US interagency collaborative conference hosted by the Partnership Strategy Office on DoD HA identified the M&E system as the way forward in implementing HA impact assessment.27 The conference included representatives from each Combatant Command (African, Central, European, Northern, Pacific, and Southern Commands), the US Agency for International Development, and the State Department’s Bureau of Population, Refugees, and Migration. Whereas the M&E system was a product of the working group discussion, it is not DoD policy at this time.

The M&E system is a management tool used by policy makers and decision makers and managers to track performance and assess the impact of programs, projects, and policy. The following 5 steps were adopted from a 10-step M&E system and outline the basic progression to effective utilization of the impact assessment model8:

1. 1. Readiness assessment

2. 2. Formulate health outcomes and goals

3. 3. Select health outcome indicators to monitor

4. 4. Gather baseline information on HN current health conditions

5. 5. Set specific targets to reach and dates for reaching them

Each step is intended to complement military operation planning and execution and provides the necessary focus and information to effectively use the impact assessment process and terminology shared by all US development agencies during HA operations.

Step 1: Readiness Assessment

The readiness assessment is an analytical approach to evaluate the DoD and HN operational capacity and partnership capability to perform HA activities in stability operations. The following questions investigate DoD and HN willingness, capability, capacity, feasibility, appropriateness, and comparative value-added advantage when HA activities are being considered in stability operations8:

1. 1. What motivates the need to include HA in stability operations? For example: mitigate existing mortality and morbidity, foreign policy, diplomacy, national security, altruism, support global health initiatives (eg, HIV/AIDS), military training, strategic communication, or strategic initiatives.

2. 2. How would health-related interventions support stability operations in the HN of interest?

3. 3. Who would benefit from the HA activities?

4. 4. Who will not benefit from the HA activities, and why?

5. 5. What is the role of the HN government and its participating agencies?

6. 6. What is the capability and capacity of the recipient community?

7. 7. Are there HN government or nongovernment agencies that produce data, collect data, or do outcome research?

8. 8. Can the HN provide technical assistance?

9. 9. Who in the DoD will use the collected data, and for what purpose?

10. 10. Are there organizations already providing relief and development in the areas of interest?

11. 11. Would the military-led HA activities complement existing HA activities?

12. 12. In which specific areas can DoD find comparative advantage for value-added HA interventions?

Step 2: Choose Outcomes

Outcomes are the results of HA activities that should be collected and reported by DoD. All of the HA activities in stability operations should begin with identifying health-related outcomes reflecting larger US strategic and previously established HN development goals. Although health outcomes are ultimately what the impact assessment model measures at the end of an HA operation, the intent of implementing this model is to focus DoD on considering outcomes first. Stakeholder and HN recipient involvement in planning and executing HA activities often is overlooked by the DoD.Reference Reaves, Schor and Burkle1, Reference Drifmeyer and Llewellyn28 Using the participatory approach early in HA planning, by first choosing mutually agreed upon desired program outcomes, can initiate strong diplomatic partnerships with recipient HNs and ensure mutually favorable HA program results.7 This process also will enhance strategic and nation-state communication and ensure that the DoD HA elements in stability operations are in-line with long-term strategic goals. In essence, the focus starts with desired health outcomes and works backward to learn which HA activities and inputs (resources) are necessary to successfully achieve those desired outcomes. Once the necessary inputs and activities to successfully implement HA activities and link them to desired outcomes are identified, the impact assessment model can be used to guide and measure results. Choosing outcomes first serves as the foundation for the remaining steps in the impact assessment model, and commanders can use outcomes as benchmarks to manage HA programs.

DoD HA activities should complement regional and global health initiatives, such as those outlined by the United Nations Millennium Development Goals (MDGs),29 the Global Fund,30 the World Health Organization (WHO) International Health Regulations,31 the Sphere Project,Reference Wilson32 the World Bank, and others.Reference Mock and Garfield24 This strategy will support broad-based internationally accepted development goals and initiatives already put in place by other governments and aid organizations. MDGs, such as World Bank and WHO development goals, have also been translated into measurable outcome indicators for assessing the impact of interventions to meet these goals. The process recommended for choosing outcomes includes the following8:

1. 1. Identify goals—use participatory approach to create consensus with HA recipients and HNs using focus groups, surveys, and interviews

2. 2. Translate health problems into positive improvements

3. 3. Create an outcome statement including 1 improvement solution

4. 4. Disaggregate outcome statement to include the following:

• Population affected

• Location

• Percent change or quantity desired in indicators

• Duration—over what period of time change is expected

Step 3: Select Indicators

Indicators are variables, both quantitative and qualitative, that are simple to use and reliably measure changes or show performance associated with an HA intervention.16 Indicators link specific HA activities with desired outcomes and are used to manage program implementation and progress, and to determine whether desired outcomes are being achieved. Indicators are the measurable entity of outcomes that drive successive HA program data collection, analysis, and reporting. Indicators are not outcomes; they are the measurable part of the outcome. Indicators are the numbers, rates, or percent changes that data collectors measure to see whether outcomes are being achieved. For example, if the desired outcome was to improve infant mortality, possible indicators to measure may include infant mortality rate, mortality rate under age 5, maternal mortality, number of deliveries attended by trained personnel, or rate of births in facilities with infant-maternal delivery services.

Indicators should have standard and accepted methods of measurement to help ensure reproducibility and facilitate comparability. Criteria for selecting outcome indicators can be guided by the Standardized Monitoring and Assessment of Relief and Transition initiative.33 This method is preferred for DoD HA because it is used by the US Agency for International Development and many of their funded nongovernmental organizations in the field.Reference Mock and Garfield24 Quantitative indicators are preferred over qualitative indicators when possible, and should be reported as a specific number (descriptive statistic) or percentage. Qualitative indicators, although valuable when used appropriately, can be subjective, circumstantial, and difficult to verify. They should not be used exclusively to assess impact or make evidence-based policy decisions.

Following the impact assessment model shown in Figure 1, indicators can be divided into categories for project outputs (implementation) and project outcomes (results). For a given HA activity, both output and outcome indicators need to be assigned and measured. An example including both indicators follows:

1. 1. Outcome desired: decrease vaccine preventable disease in population of interest

• a. Outcome indicator: percent (reduction) in vaccine-preventable specific diseases in population

1. 2. Output: vaccines administered to population

• a. Output indicator: number of vaccines given to recipient population (percentage of vaccine coverage in population)

Choosing indicators is best done in each stability operation to match specific HN, regional, or global health needs by involving HA recipients and stakeholders in a participatory consensus process. When applicable, it is possible to use standardized indicators such as those provided by the Standardized Monitoring and Assessment of Relief and Transition initiative,33 UN MDGs, World Bank,34 WHO, Food and Nutrition Technical Assistance Project,35 and others.

Step 4: Collect Baseline Data

The collection of baseline data precedes HA activities and serves as the initial data for future comparison with impact assessment indicator measurements during and after HA activities. Baseline data are necessary to predict health needs before HA operations, to choose indicators, and to follow outcome trends. It is recommended that the DoD use baseline data sources from health surveillance that are already standardized, collected, reported, and made available by HNs, humanitarian aid organizations, or health governing agencies. Baseline DoD-directed data collection may be difficult because of 1-time, short-duration HA operations, biased recipient population representation, and lack of baseline population statistics. Those receiving DoD HA services may not be representative of the HN general population, and health conditions in this population subset may change over time. Baseline collection and analysis of data by designated HN institutions or aid organizations working in the regions of interest are recommended because of their geographic location and the ability to facilitate the participatory approach and reinforce HA program continuity and sustainability after DoD HA activities have been successfully transferred to the HN and aid community.7 This approach will simplify administrative burden and ensure agencies continue to collect data on DoD HA interventions over time. In settings where follow-up DoD HA operations are repeated, active HN data collection collaboration will ensure continuous impact assessment and mission-specific comparability between multiple operations. When important baseline data are missing or unavailable, the DoD may then consider a data collection strategy incorporating initial HA planning and HN site surveys to capture the recipient population of interest and facilitate early planning, assessment of feasibility, and stakeholder participation.

A combination of data collection methods is recommended and will more reliably produce consistent data. As HA activities in stability operations continue, indicators, data collection sources, and data collection techniques will evolve; however, no 2 HA operations will be equal in regard to using data collection techniques for effective impact assessment.

Step 5: Establish Targets

Targets as an epidemiological term refers to aspired outcomes that are explicitly stated using quantifiable levels of indicator changes established by predicting the desired level of improvement of an indicator from baseline indicator measurements during a specific period of time. They should be achievable within the scope and inherent limitations of the culture, environment, security, and so forth, in that DoD lessons learned, resources, finances, and HN existing capacity and goals must be considered when predicting performance targets. The DoD should set interim HA targets that reflect traditional short-term, 1-time amendable HA interventions typical of DoD HA activities in the context of larger long-term strategic goals. Targets should be numerical and should represent specific indicators, include the desired amount of change with a range of acceptable outcomes and the time interval for expected change to occur, and reflect HA team progress.

APPLYING THE IMPACT ASSESSMENT MODEL BY EXAMPLE

Table 2 provides a public health example using the previously presented modified M&E guidelines for selecting desired HA outcomes and moves sequentially to identify indicators, collect data, and form a target. The example uses MDG No. 6 (“combat HIV/AIDS, malaria, and other disease”) to highlight the importance of linking specific DoD HA goals with broader agreed-upon global health goals when designing a HA operation. The outcome goal and indicator to measure are standardized specific to the United Nations Millennium Declaration.29 The baseline data and target are factitious for this example. In practice, the baseline data would be collected for the population of interest and the target set using knowledge of available resources, finances, and HN existing capacity.

TABLE 2 Example of Modified Monitoring and Evaluation (M&E) Steps 2–5 Using a Public Health Example

Once desired stakeholder outcomes and targets are established to predict HA inputs and baseline data are collected, both output and outcome indicators can be measured to assess the impact of HA activities following the impact assessment model. Figure 2 shows how the impact assessment model can be used to link the methods of HA program M&E and assessment of HA impact using a public health example with an outcome goal of reducing infant mortality.23

FIGURE 2 Use of the impact assessment model assessing aid impact with a public health example. ORT = oral rehydration therapy. Modified from referenceReference Wilson32.

To assess HA impact, past and present data are compared for trends. Data comparisons over time—including baseline, intermediate, and postintervention measurements—are important to report when assessing HA impact. This allows commanders to link HA program outcomes with preestablished targets when comparing HA effectiveness. As more measurements are taken and more indicator data are collected, the validity, reliability, and confidence in trend changes can be strengthened.

Conclusions

Impact assessment is a continuous process that links HA program implementation with results starting from the beginning of HA planning and continues after DoD HA activities are transferred to the HN and aid community. The impact assessment model draws on the strengths of DoD resources during health sector HA in stability operations and provides a process for linking health sector activities with desired program results. The key to impact assessment is measurement of outcome indicators, which includes the correct identification, collection, and analysis of data both specific to HA project interventions and the broader stability-strategic goals. Although the health sector in HA operations was emphasized, this model is easily adaptable at the larger military mission level, both regionally and globally. As the successful use of the impact assessment model continues, lessons learned and data collected should facilitate results-based policy decisions in DoD HA.

Authors' Disclosures The authors report no conflicts of interest.