Health policy decisions are increasingly based on health technology assessments (HTAs). HTA is a systematic approach to evaluate the properties, effects, and impacts of health technologies or interventions (1). However, while most HTAs focus on clinical medicine and pharmaceuticals, HTAs of public health interventions (PHIs) are still rare (Reference Draborg, Gyrd-Hansen, Poulsen and Horder2;Reference Lavis, Wilson and Grimshaw3).
HTA methods to evaluate public health interventions may differ substantially from clinical/medical HTAs. Assessing complex interventions such as those in the field of public health is associated with a range of challenges, such as very heterogeneous evidence due to the variety of methodological characteristics, and the diversity of populations, interventions and intervention components, comparisons, and outcomes and outcome measurements (Reference Burford, Lewin, Welch, Rehfuess and Waters4). Due to their complexity, public health interventions are, for example, often evaluated using nonrandomized study designs (Reference Petticrew, Anderson and Elder5–Reference Hartling, Bond, Santaguida, Viswanathan and Dryden7). The typical methodological decisions made when carrying out HTAs, such as the choice of electronic databases or risk of bias assessment, are, therefore, not always applicable for HTAs on public health interventions (Reference Akers, Aguiar-Ibáñez and Baba-Akbari8–Reference Gerhardus, Oortwijn and van der Wilt10), for example. Therefore, it is challenging to choose which methods to use for such HTAs.
There are only few HTA agencies that provide standardized, formalized methods specific for public health interventions and they exhibit surprising heterogeneity (Reference Mathes, Antoine and Prengel9). Until now, the methods applied in HTAs on public health interventions have not been systematically assessed.
Objectives
The aim of this study was to provide an overview of the methodological characteristics and compare the assessment methods that have been used in HTAs of PHIs. More specifically, our purpose was to cast some light on methodological approaches to address challenges in HTAs of PHIs.
Methods/Design
The detailed methods are presented in our protocol (Reference Mathes, Willms and Polus11). As no outcome of direct patient or clinical relevance is assessed in this work, the protocol was not registered in PROSPERO.
Searches
We systematically searched the webpages of members of the International Network of Agencies for Health Technology Assessment (INAHTA), Health Technology Assessment International (HTAi), and the European Network for Health Technology Assessment (EUnetHTA), and screened the full lists of all published HTAs. Between June and July 2017, one reviewer performed the searches and preselected all potentially relevant titles. References were managed with EndNote X7.
Inclusion Criteria
Two reviewers independently screened potentially relevant full text reports according to the following inclusion criteria: (i) Full HTA report as defined by INAHTA (Reference Merlin, Tamblyn and Ellery12); (ii) Assessment of a public health intervention; (iii) Publication date: 2012 to 2016; and (iv) Language: English, German, Spanish, French. We excluded all literature review-based HTAs using accelerated (e.g., rapid reviews) or abbreviated HTA/systematic review methods, overviews of reviews (or umbrella reviews), scoping reviews, mini-HTAs, etc., and protocols (Reference Merlin, Tamblyn and Ellery12;Reference Page, Shamseer and Altman13).
In this review, we considered only population-based interventions on health promotion and interventions for primary prevention of noncommunicable (e.g., cardiovascular diseases, diabetes, and injuries) or infectious diseases to ensure consistent study selection (14). We excluded HTAs on screening and vaccination because these require special evaluation methods (e.g., diagnostic accuracy studies or modeling) (15). Furthermore, we decided only to include HTA reports published from 2012 to 2016 to have a full coverage of 5 years. HTA reports of all countries were eligible for inclusion.
Data Extraction and Quality Assessment
Two reviewers independently extracted the data, except for ethical aspects, where the data were extracted by one reviewer and verified by a second reviewer. Data extraction included details on the methods applied for the domain's effectiveness/safety, as well as for economic, social, cultural, ethical, and legal aspects. In case of disagreement, the reviewers discussed the problematic cases and consulted with a third reviewer. We piloted the data extraction forms a priori using HTA reports published before 2012. We did not assess the quality of HTA reports because our focus lay on exploring methodological features of the HTAs.
Data Synthesis and Analysis
We tabulated the information retrieved from the reports for each domain using Microsoft Excel (2010). We planned to describe dichotomous and nominal variables using absolute numbers and percentages, and to show means and standard deviations for metric variables. Due to the small sample size, however, we did not perform statistical analyses but instead report only absolute numbers. We also planned to perform subgroup analyses according to public health intervention focus, healthcare system, target audience, and evaluation level. This was, however, not feasible due to the small study sample.
Postprotocol Changes
Data extraction was performed independently as reported in the protocol, except for the ethical aspects as detailed above.
Results
We screened the websites of 127 different HTA organizations and found 125 potentially relevant HTA reports published by thirty-four institutions (see Supplementary File 1). We excluded the majority of HTAs because they did not comply with our definition of an HTA (see Supplementary File 2 for a list of excluded HTA-reports). We included ten HTAs from four countries/organizations (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–Reference Bambra, Hillier and Cairns25), including five HTAs from the National Institute of Health Research in the United Kingdom, three HTAs from the German Institute of Medical Documentation and Information, and one HTA each from the Health Information and Quality Authority in Ireland and the Institute for Clinical and Economic Review in the United States. The selection process is depicted in Figure 1. The included HTA reports and their main characteristics are listed in Tables 1 and 2. More detailed characteristics can be found in Supplemental File 3. In the following, the study IDs detailed in Table 1 are used to refer to the HTA reports.
Fig. 1. Flow chart.
Table 1. Main Characteristics
Table 2. Assessed Aspects
All ten HTA reports assessed the health effectiveness and cost effectiveness of the interventions. Seven documents also investigated social/cultural aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23), four considered legal aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Fröschl, Brunner-Ziegler and Wirl17;19;Reference Tice, Chapman and Shore20), and four reports also discussed ethical aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–19).
The scope and specificity of HTA objectives differed across reports. Whereas some narrowed their objectives to for example, assessing effectiveness (and cost-effectiveness) of the intervention (e.g., Balzer et al., 2012;, Korczak et al., 2012) (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Fröschl, Brunner-Ziegler and Wirl17), others provided an extensive list of detailed objectives extending to specific aspects such as acceptability of the intervention (e.g., HIQA, 2014; Tappenden et al., 2012) (19;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22).
Bee et al.; 2014 (Reference Bee, Bower and Byford21); and O'Mara-Eves et al.; 2013 (Reference O'Mara-Eves, Brunton and McDaid24); involved stakeholders to aid the review process. Five reports used, modified, or developed a framework to assist in conducting the review, for example, to conceptualize how the intervention may work or may be modified (Reference Korczak, Kister and Krause-Girth18;Reference O'Mara-Eves, Brunton and McDaid24), to scope or organize possible intervention types (Reference Brown, Todd and O'Malley23;Reference Bambra, Hillier and Cairns25), or to assess possible health-related quality-of-life outcomes (Reference Bee, Bower and Byford21). It was, however, not always feasible to identify to what extent and in what way the framework was finally used in the actual conduct of the review. O'Mara-Eves et al., 2003 (Reference O'Mara-Eves, Brunton and McDaid24), additionally conducted a process evaluation using a previously developed data extraction tool for public health interventions.
Health Effectiveness/Safety
The reports searched a range of 3–32 databases. Two reports included only randomized controlled trials (RCTs) to assess effectiveness/safety in their systematic reviews (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22), whereas the rest additionally included nonrandomized study designs, for example, cohort studies, nonrandomized trials, controlled before–after and interrupted time series studies. Five reports used additional sources, such as grey literature or contacting experts/key contacts to complement their search strategy (Reference Tice, Chapman and Shore20;Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23;Reference O'Mara-Eves, Brunton and McDaid24;Reference Bambra, Hillier and Cairns25).
The risk of bias tool most often used was the Cochrane risk of bias tool, but some reports (additionally) used other tools or a combination of tools to meet their needs. These included, for example, the Effective Public Health Practice Project Quality Assessment Tool for Quantitative Studies (EPHPP) (19;Reference Brown, Todd and O'Malley23), the EPPI Centre tool for quality assessment (Reference O'Mara-Eves, Brunton and McDaid24), or Oxford Centre for Evidence-based Medicine Levels of Medicine (Reference Korczak, Kister and Krause-Girth18). Froeschl et al.; 2013 (Reference Fröschl, Brunner-Ziegler and Wirl17); developed their own tool, and O'Mara-Eves et al.; 2013 (Reference O'Mara-Eves, Brunton and McDaid24); modified the Cochrane risk of bias tool for their purposes. One report integrated the risk of bias assessment in the data extraction, but did not report the tool being used (Reference Bambra, Hillier and Cairns25).
Four publications (Reference Tice, Chapman and Shore20;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22;Reference Brown, Todd and O'Malley23;Reference Bambra, Hillier and Cairns25) considered context/setting in their assessments. Brown et al.; 2016 (Reference Brown, Todd and O'Malley23); and Bambra et al.; 2015 (Reference Bambra, Hillier and Cairns25); used the “methodological tool for the assessment of the implementation of complex public health interventions in systematic reviews” (Reference Egan, Bambra, Petticrew and Whitehead26) for their assessment, adapted for obesity interventions in the case of Bambra et al.; 2015 (Reference Bambra, Hillier and Cairns25). Two reports narrowed down the scope of their review to the national context (Reference Tice, Chapman and Shore20;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22), and Tice et al.; 2016 (Reference Tice, Chapman and Shore20); also applied a value assessment framework and a stakeholder panel to generate contextual information. Bambra et al.; 2015 (Reference Bambra, Hillier and Cairns25; also separated results according to the international or the UK context.
We also assessed whether the reports considered the applicability or generalizability of the results. One report (19) discussed applicability aspects but did not use a specific methodological tool to systematically assess these aspects. In a separate section, they discussed the “applicability of the results in an Irish context” and “applicability of other studies,” which are also elaborated on in the discussion section of the HTA report. One other HTA, Bambra et al., 2015 (Reference Bambra, Hillier and Cairns25), partly considered applicability of the evidence in their discussion.
Three reports from the United Kingdom (Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23;Reference Bambra, Hillier and Cairns25) assessed intervention integrity. Bee et al., 2014 (Reference Bee, Bower and Byford21), detailed as explicit objectives “to explore all available data relating to the acceptability of community-based interventions […]” and “to assess key factors influencing the acceptability of and barriers to the delivery and implementation of community-based interventions.” They conducted one large search, which incorporated all objectives of the HTA, and included acceptability studies that were either quantitative/qualitative or mixed-method approach studies. Bambra et al., 2015 (Reference Bambra, Hillier and Cairns25), and Brown et al., 2016 (Reference Brown, Todd and O'Malley23), assessed delivery fidelity as well as sustainability of the interventions with the same tool used for their context/setting assessment (Reference Egan, Bambra, Petticrew and Whitehead26). None of the other reports assessed sustainability of the intervention.
Seven reports that (at least partly) conducted meta-analyses assessed statistical heterogeneity by means of the I2 statistics or other statistical methods. One did not pool data due to high heterogeneity (Reference Tice, Chapman and Shore20). One report combined quantitative and qualitative studies in a mixed-method approach (Reference Bee, Bower and Byford21), and four reports explicitly conducted subgroup analyses, for example, for different populations or settings (19;Reference Bee, Bower and Byford21;Reference O'Mara-Eves, Brunton and McDaid24;Reference Bambra, Hillier and Cairns25).
Economic Aspects
Of the ten included HTA reports, the majority conducted systematic reviews of economic evaluations. Only one conducted a primary economic evaluation in addition to a systematic review of economic evaluations with the purpose to inform the primary economic evaluation (19). One report conducted a nonsystematic review of economic evaluations (Reference Tice, Chapman and Shore20).
HIQA 2014 (19) conducted a cost-utility analysis as a primary economic evaluation and, additionally, a post-hoc budget-impact analysis. The comparator was routine care, which reflects the societal analysis perspective. HIQA 2014 (19) considered only direct medical costs. They did not report the valuation of outcomes. They used a Markov modeling approach and performed a probabilistic sensitivity analysis. They presented the results narratively as well as with graphic aids, such as tables, a cost-effectiveness plan, and an acceptability curve.
All reports except Tice et al. 2016 (Reference Tice, Chapman and Shore20) conducted systematic reviews of economic evaluations and included full economic evaluation types (cost-effectiveness analyses, cost-utility analyses, and cost-benefit analyses). Other included economic evaluation types were cost-consequences analyses (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Korczak, Kister and Krause-Girth18;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22), and cost-minimization studies (Reference Bambra, Hillier and Cairns25). Four publications did not specify which studies they included (Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23–Reference Bambra, Hillier and Cairns25).
Reports differed on whether the search strategy for economic evaluations was integrated into the overall generic search or whether they conducted a separate search in addition to the generic search. Korczak et al. 2012, Balzer et al. 2012, Froeschl et al. 2013, Brown et al. 2016, O'Mara-Eves et al. 2013, and Bambra et al. 2015 (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–Reference Korczak, Kister and Krause-Girth18;Reference Brown, Todd and O'Malley23–Reference Bambra, Hillier and Cairns25) integrated the economic search in the overall search strategy on effectiveness, and in addition to the generic databases, searched one economic database (i.e., NHS Economic Evaluation Database). HIQA 2014 (19) and Bee et al. 2014 (Reference Bee, Bower and Byford21) conducted separate searches for the economic evaluations; in one report this was not clearly specified (Reference Tappenden, Campbell, Rawdin, Wong and Kalita22). In four reports, both the title-abstract screening and the full-text screening were conducted by two persons independently (title-abstract: Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Korczak, Kister and Krause-Girth18;Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23; full text: Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Fröschl, Brunner-Ziegler and Wirl17;Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23). Three and four reports, respectively, did not provide clear information regarding the title-abstract screening or the full text screening (title-abstract: Reference Fröschl, Brunner-Ziegler and Wirl17;19;Reference O'Mara-Eves, Brunton and McDaid24; full-text: Reference Korczak, Kister and Krause-Girth18;19;Reference Tappenden, Campbell, Rawdin, Wong and Kalita22;Reference O'Mara-Eves, Brunton and McDaid24).
Of the ten reports, four presented cost data as reported, two converted the currency (inflated and in Euro). In one case, this was not clear, and three reports did not include any economic evaluation studies. All presented the results exclusively narratively. All HTAs that identified studies in their searches assessed the quality of the included studies. It was only possible in one case to determine an explicit effect of the economic evaluation on the HTA overall results/decision (19); for most, this remained unclear or was not applicable due to missing studies.
Social and Cultural Aspects
Seven HTAs addressed social and/or cultural aspects to a different extent (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23). Three reports used a theoretical framework to conceptualize the social/cultural aspects. HIQA 2014 (19) used the EUnetHTA Core Model, which addresses eight different issues of patients and social aspects. Tice et al. 2016 (Reference Tice, Chapman and Shore20) used the “care value framework” that addresses significant benefits or disadvantages to the patients, their caregivers, the delivery system, other patients, or the public next to benefits or disadvantages in relation to health effectiveness. In a first step, Bee et al. 2014 (Reference Bee, Bower and Byford21) conceptualized health-related quality of life outcomes through stakeholder consultations, the main outcome of the HTA. The other four HTAs did not report a framework.
All seven HTAs used a systematic literature search and hand searches to identify studies that assess social and cultural aspects. Except for HIQA 2014 (19), all reports integrated their search strategy into the overall search for the HTA. HIQA 2014 (19) did not report their methods in detail but referred to the EUnetHTA Core Model, which suggests a separate systematic search for patient-related issues and social aspects. Brown et al. 2016 (Reference Brown, Todd and O'Malley23) limited their search to controlled studies. HIQA 2014 (19) and Bee et al. 2014 (Reference Bee, Bower and Byford21) combined review methods (systematic literature searches) with qualitative methods, for example, semi-structured interviews, expert interviews, and policy roundtables. Quality assessment of the included studies was performed in five HTAs (Reference Korczak, Kister and Krause-Girth18–Reference Bee, Bower and Byford21;Reference Brown, Todd and O'Malley23) applying different assessment tools. All results on social and cultural aspects were reported narratively. The findings on social and cultural aspects had implications for the HTA, as the reports integrated the results in the discussion and recommendations.
Legal Aspects
Four reports addressed legal aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Fröschl, Brunner-Ziegler and Wirl17;19;Reference Tice, Chapman and Shore20). Each of these used a different method. Balzer et al. 2012 (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16) conducted an additional hand search of the literature to retrieve information on legal aspects, Froeschl et al. 2013 (Reference Fröschl, Brunner-Ziegler and Wirl17) summarized information that was found in the studies identified through the overall systematic literature search for the HTA. In HIQA 2014 (19), a legal analysis of the technology was performed by an additional team from the faculty of law at a higher-education institution. Tice et al. 2016 (Reference Tice, Chapman and Shore20) included legal aspects from the perspectives of the government, payers, purchasers, patients, and vendors combining a literature review, semi-structured interviews, and policy roundtable discussions. All results were reported narratively.
Ethical Aspects
Four reports self-identified as addressing ethical aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–19). Ethical issues were sometimes addressed along with other aspects, legal (Reference Fröschl, Brunner-Ziegler and Wirl17) or social and legal (Reference Korczak, Kister and Krause-Girth18), without clear differentiation between the different issues. Two reports dedicated several paragraphs solely to ethical aspects, although they discuss and analyze ethics in close association with social (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16) or legal (19) aspects. Three reports (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16–Reference Korczak, Kister and Krause-Girth18) used a systematic literature search to identify relevant information, with only two specifically searching for literature addressing ethical aspects (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16;Reference Fröschl, Brunner-Ziegler and Wirl17). Findings were narratively synthesized, with one report pointing out that only those points from the literature were described that were perceived as relevant by the authors (Reference Fröschl, Brunner-Ziegler and Wirl17). One report applied a theoretical framework, principlism (Reference Beauchamp and Childress27), to identify and discuss ethical issues without conducting an additional literature search (19). In all reports, the outcomes of the analyses were considered in the conclusion at least to a certain extent.
Discussion
Effectiveness, Safety and Economic Aspects
The HTAs used a range of approaches to deal with the challenges of evaluating the effectiveness/safety of public health interventions. Most reports adapted their methods to a certain extent, for example, by applying a different risk of bias assessment or by including a variety of nonrandomized study designs. While some merely searched in several databases, others also used alternative sources to search for studies, such as grey literature or expert and stakeholder consultations. A recent methodological case study confirms the effectiveness and better value of supplementary searches compared with mere database searching in public health (Reference Cooper, Lovell, Husk, Booth and Garside28). Others broadened their scope, included different research objectives, and addressed more than just (economic) effectiveness aspects by considering the context/setting of the HTA results, applicability and/or aspects such as implementation and acceptability of the intervention. However, few HTAs assessed these aspects systematically. It is indeed interesting that despite the heterogeneity of methods chosen, most reports considered nonrandomized study designs as recommended by HTA guidance on public health interventions (Reference Mathes, Antoine and Prengel9). This confirms that RCTs are often not feasible or unethical for the assessment of public health interventions (Reference Polus, Pieper and Burns29).
Overall, due to the increasing awareness of the costs of health services, it seems important that HTA reports in the context of public health give higher priority to economic analyses. This should be reflected above all in the reporting quality and methods used in the economic evaluation. Taking into account that costs of public health interventions are quite high because population-based target groups are typically large, one could expect that HTA authors had estimated the financial consequences of the interventions, for example, by budget impact analysis. However, only one HTA performed a primary economic evaluation and budget impact analysis (19).
Furthermore, in most cases, the economic evaluation was not discussed in the context of other domains of the HTA such as effectiveness/safety, and its influence on final recommendations was not explicitly clarified. In addition, it seems appropriate to recommend considering a more accurate application of existing guidance for conducting systematic reviews of economic evaluations (Reference Mathes, Walgenbach, Antoine, Pieper and Eikermann30). Approaches for systematically reviewing the economic literature were very heterogeneous, and standards were often not fulfilled. In four of the nine HTA reports conducting systematic reviews of economic evaluations, clear information about full-text screening (e.g., reviewers involved) was missing; four reports did not specify which study types were included. This is remarkable, because this procedure is crucial to ensure the quality of the review process.
Social, Cultural, and Legal Aspects
Social and cultural norms influence how people perceive a health issue, as well as their acceptance of an intervention and its implementation (Reference Lysdahl, Mozygemba and Burns31). This can modify the effectiveness of an intervention between different groups. Especially in public health interventions, social, cultural, and often legal aspects play a crucial role. Despite this, we found only seven reports that addressed social or cultural aspects, and even fewer that addressed legal aspects.
Within these reports, we found wide variation regarding the frameworks and methods used. In all HTAs, a systematic literature search was the basis for assessing social and cultural aspects. Most HTAs used a second or even third method in addition. For assessing legal aspects, each HTA used a different method. A consensus or codification of methods is still missing in these areas. This might at least partly explain why only few HTAs assess those aspects, despite their relevance for public health interventions. Whenever social, cultural, and legal aspects were assessed, the results had explicit implications on the HTA overall results, which further underlines their relevance.
Ethical Aspects
The majority of HTA reports addressing public health interventions did not discuss ethics. In that respect, our findings are similar to studies within other contexts (Reference DeJean, Giacomini, Schwartz and Miller32). What is discussed under the label of ethics is quite heterogeneous. Korczak et al. 2012 (Reference Brown, Todd and O'Malley23), for example, only describes empirical data such as risk factors, while Balzer et al. 2012 (Reference Balzer, Bremer, Schramm, Lühmann and Raspe16) discusses the implications of relevant legal cases, and the authors of HIQA 2014 (19) elaborate on duties and moral conflicts arising from certain bioethical principles relevant to their context of interest. The methods used to gather relevant information also vary from systematic searches to the application of theory (principlism). This heterogeneity does not come as a surprise as there is an impressive variety of methods (and implied conceptualizations of what an ethical issue is) proposed for ethics assessments (Reference Hofmann33;Reference Assasi, Schwartz, Tarride, Campbell and Goeree34).
It has been argued that this methodological heterogeneity is not necessarily problematic as various approaches to ethical analysis arrive at similar results (Reference Saarni, Braunack-Mayer, Hofmann and van der Wilt35). However, some of the reports included here arguably do not even address ethical aspects, at least when ethics is understood as an endeavor to move beyond what is to what should be, rather than as equivalent to a legal analysis. Furthermore, certain methodological choices, for example not specifically searching for ethics literature but using effectiveness or safety studies to identify ethical issues, were questionable (Reference Droste36). Accordingly, it seems to be of importance for institutions commissioning HTAs also in public health contexts to provide clearer guidance on how ethics assessments should be conducted, and possibly also on the question under what circumstances these assessments are necessary in the first place (Reference Grunwald37). This is particularly true insofar as outcomes of ethics analyses affected the final conclusions of the HTA reports.
Addressing Complexity
Very few used, modified, or developed theoretical frameworks to aid their HTA development process, despite the need for sound theoretical understanding of how a complex intervention causes change (Reference Craig, Dieppe and Macintyre38). It often remained unclear how much the framework guided the process. Others simply discussed the different aspects in separate sections or chapters. This result is not surprising, as detailed descriptions of most process steps are missing even in existing guidance (Reference Mathes, Antoine and Prengel9). None of the reports explicitly mentioned having followed guidance for HTAs on public health interventions as identified by Mathes et al. (Reference Mathes, Antoine and Prengel9). Brown et al. 2016 (Reference Brown, Todd and O'Malley23), however, stated to have used the EPHPP tool as “recommended by the Cochrane Public Health Review Group”. Two recent HTAs conducted by the Public Health Research (PHR) program of the National Institute for Health Research (NIHR) in the UK explicitly mention the complexity of public health interventions in the context of their research (Reference Brown, Todd and O'Malley23;Reference Bambra, Hillier and Cairns25). Both have used the “methodological tool for the assessment of the implementation of complex public health interventions in systematic reviews” (Reference Egan, Bambra, Petticrew and Whitehead26) to extract data on different aspects guided by the tool to assess intervention effectiveness.
Despite having applied a heterogeneous set of methods and approaches, most HTAs differed from traditional medical/clinical HTAs in that they introduced new methods and addressed additional aspects, trying to tackle the complexity of the intervention. While the German HTAs tend to use a single standardized, inflexible approach to assess effectiveness/safety of the intervention, the UK HTAs more often applied different methods to approach effectiveness and related aspects, such as context/setting or applicability. However, there seems to be no systematic, standardized approach and the methods are heterogeneously applied. Surprisingly, HTAs from one organization did not necessarily use the same methods although we could see an overall tendency, for example, for assessing ethical or social/legal aspects within an organization (Germany), or for assessing further aspects and processes, such as context or implementation/acceptability issues (United Kingdom). Due to the small sample included we were unable to identify patient or PHI characteristics that would explain the heterogeneity of methods. Complex public health interventions differ from one another in many ways, leading to various problems and research questions/aspects to be addressed, which may lead to the need to adapt approaches dealing with the different individual challenges. A consistent overall methodology would, however, be an advantage for decision makers and other consumers of HTAs in terms of understanding as well as in increasing credibility and facilitating the conduct of HTAs of public health interventions.
A recent European Union (EU)-funded project developed guidance specifically to address the challenge of evaluating complex intervention in HTAs (Reference Gerhardus, Oortwijn and van der Wilt10). In future research, it would be interesting to see whether approaches to conducting and integrating systematic assessments of the different aspects important in HTAs evaluating public health interventions will become visible.
Limitations
We found few HTAs assessing public health interventions, and the majority of these were conducted by only two organizations, one each from Germany and the United Kingdom. We did not include a broader scope of public health interventions to allow us to distinguish included HTAs from medical/clinical HTAs, which may have partly led to such a small study sample. Furthermore, the decision as to what constitutes a public health intervention is not straightforward. Our strict inclusion criteria, which have served to comply with a higher quality standard, may also have contributed to this. We excluded further HTA reports due for language reasons, including reports from Lithuania, Norway, and Sweden that were only published in the respective national language. In addition, the list of HTA agencies might be incomplete. This work may, therefore, not encompass a representative sample of all HTAs on public health interventions. Furthermore, our search was limited to the years 2012–16.
In conclusion, despite the limitations listed above, the results show that very few HTAs have yet been conducted in the field of public health. One reason may be that economic analyses of public health interventions tend to be more difficult due to the necessary long study periods. This may have prevented organizations from conducting economic evaluations in the first place. We found that some HTA organizations flexibly adapted the methods according to the specific complexity of the public health intervention assessed. Such need for flexibility may pose a further barrier for some organizations to conduct HTAs in this field. This may also partly explain why we found so few HTAs.
Health policy all over the world, however, continues to implement public health legislation and programs, seemingly without using systematic, evidence-based approaches similar to those commonly used in the traditional fields covered by HTAs, that is, clinical medicine and pharmaceuticals. Considering the importance, scope, and long-term benefits (or harms) as well as the potentially high budget impact of public health interventions in the context of limited resources, these findings should motivate the further research and development as well as international exchange of HTA methodology for public health interventions.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000515
Conflicts of interest
The authors declare that there are no conflicts of interest.
