The majority of health technology assessments (HTAs) still focus on clinical medicine, particularly on pharmaceuticals (Reference Draborg, Gyrd-Hansen, Poulsen and Horder1), while HTAs on public health interventions (PHI) are rarely conducted (Reference Draborg, Gyrd-Hansen, Poulsen and Horder1). In 2002, David Sackett criticized the use of preventive interventions without an evidence base (Reference Sackett2). A survey conducted in five countries in 2010 found that only 5 percent of HTAs focused on public health (Reference Lavis, Wilson and Grimshaw3).
Conducting HTAs of PHIs poses some challenges compared with medical interventions. Randomized controlled trials (RCTs) are often not available in the field of public health as they are usually difficult to conduct (Reference Petticrew, Chalabi and Jones4). PHIs, such as the implementation of a school nurse, are highly complex, The standardization of interventions (e.g., nurses), various intervention components (e.g., medical and psychological), participants (e.g., different age groups), contextual factors (e.g., number of general practices nearby), and the number and variability of outcomes add to this complexity (Reference Espallargues, Pons, Almazan and de Sola-Morales5). As a result, PHIs often rely on study designs that are not at the top of the evidence hierarchy.
Our objective was to provide an overview of the existing methodological guidance on HTAs of PHIs. Furthermore, we analyzed the similarities and differences between the methodological recommendations.
METHODS
We systematically searched the Web pages of international organizations for systematic reviews and HTA organizations. The HTA organizations were identified using the member lists of the HTA umbrella organizations: the International Network of Agencies for Health Technology Assessment (INAHTA), Health Technology Assessment International (HTAi), and the European Network for Health Technology Assessment (EUnetHTA). In total, the Web pages of 135 organizations were screened. Furthermore, we searched the Web sites of Cochrane, the Centre for Reviews and Dissemination (CRD), and the Joanna Briggs Institute (JBI).
Searches were performed in March and April of 2015. We contacted all HTA organizations by e-mail to capture unpublished documents not available on the Web sites. We obtained contact information from the Web pages of INAHTA, HTAi, and EUnetHTA. We sent the initial e-mail on March 18, 2015, and a reminder on April 10, 2015. Replies were accepted until May 1, 2015.
All potentially relevant documents were screened according to the following a priori defined inclusion criteria (see Figure 1): (i) Methodological guidance for the preparation of HTAs for PHIs (handbooks, manuals, guidelines, etc.); (ii) Language: English, Spanish, German, Italian.
Figure 1. Flow-diagram of guidance selection.
We focused on methodological guidance for the preparation of research synthesis to evaluate effectiveness. We did not consider the economic, legal, or organizational aspects of a PHI. Documents focusing on evaluation methods other than research synthesis (e.g., surveys) were not considered. We assumed that the challenges of HTAs of PHIs might require fundamentally different approaches to overcome. Each step in the preparation process of an HTA (e.g., literature search) is always interrelated with all other steps conducted in the HTA. We, therefore, focused on manuals that described the entire preparation process. We did not search medical databases, as it is very unlikely that scientific papers provide detailed methodological guidance on the entire preparation process. We included documents on prevention, screening, and vaccination if described in the context of population health analysis. Only the most recent document was included if different versions of the same document existed. Two reviewers independently screened the identified publications according to the inclusion criteria. Different judgements on inclusion were resolved in a discussion until a consensus was reached.
We prepared and piloted standardized tables for data extraction. All data describing the methodology for research synthesis to evaluate the effectiveness of PHIs were summarized. Data were extracted on a priori defined aspects (see headings in Table 1 and Table 2). We extracted data specific for HTAs of PHIs. Organizational aspects, general scientific descriptions, general methods for systematic reviews, and recommendations for reporting were not extracted. We also excluded specific review types (e.g., review of reviews, rapid reviews). The wording in the included documents was copied as closely as possible to avoid interpretation bias. The data were extracted by one reviewer and checked by a second reviewer to ensure that all relevant information was captured and to guarantee the accuracy of data extraction. Discrepancies were resolved in discussion until a consensus was reached.
Table 1. Information on Definition of Public Health and HTA Products
NR, not reported.
Table 2. Methodology for HTAs of Public Health HTs
NR, not reported; HT, health technology; PICO, Patients, Intervention, Comparison, Outcome; PROGRESS, place of residence, race/ethnicity/culture/language, occupation, gender/sex, religion, education, socioeconomic status, and social capital.
RESULTS
Literature Search
The Web page search and e-mail inquiries (response rate 70 percent) of the 135 HTA organizations resulted in sixty-five potentially relevant publications provided by forty-three HTA organizations (some organizations provided more than one document). Furthermore, we searched the Web pages of Cochrane and CRD and identified two further potentially relevant publications. Of the forty-five organizations (43 HTA organizations plus Cochrane and CRD), forty-one organizations provided no eligible publications (63 excluded publications, list of excluded publications see Supplementary Material 1). Four organizations (four publications) were finally included in the analysis (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9) (Table 1). One manual was only available in German (Reference Haas, Breyer, Knaller and Weigl10). The selection process is illustrated in Figure 1.
Two publications were provided by national HTA organizations (Gesundheit Österreich [GOEG], National Institute for Health and Care Excellence [NICE]) (7;Reference Haas, Breyer, Knaller and Weigl8) and two were provided by international systematic review organizations (Cochrane Collaboration, CRD)), which develop methods to gather research evidence and publish HTAs (Reference Armstrong and Waters6;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The details on the methods are presented in Tables 1 and 2.
Scope/Definition of PHIs
The scope of public health is defined by three organizations. Two organizations use their own definitions and one is based on the World Health Organization's (WHO) definition of health promotion (7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9;Reference Haas, Breyer, Knaller and Weigl10). The holistic view of health in the area of public health is mentioned by two organizations (7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). These two organizations have a wide scope, including prevention, health protection, and health restoring and focus on population rather than individuals (7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). One organization focuses exclusively on health promotion (Reference Haas, Breyer, Knaller and Weigl8).
Review Type(s) (HTA Products)
The review types differ between the organizations. Cochrane describes the preparation of systematic reviews of effectiveness studies (Reference Armstrong and Waters6). The CRD also focuses on the preparation of systematic reviews of effectiveness studies, but in the manual it mentions that researchers may find realist synthesis useful (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Both organizations consider including qualitative studies, depending on the research question (e.g., theoretical underpinning, description of patient experience). The two national organizations produce different products depending on the research question (7;Reference Haas, Breyer, Knaller and Weigl8).
GOEG suggests the use of systematic reviews of effectiveness studies, realist reviews (Reference Armstrong and Waters6), or the interactive domain model (based on underpinnings, understanding of the environment and practice). These approaches are used separately or in combination. NICE has a broad spectrum of HTA products, including reviews of effectiveness studies, reviews of reviews, reviews of cost-effectiveness studies, and reviews of epidemiological studies (7). In general, one effectiveness review is supplemented by other products.
Planning the Review
Three organizations recommend starting with the development of a conceptual framework and a definition of scope (Reference Armstrong and Waters6;7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9), including the formulation of the review question(s). All organizations suggest using the PICO scheme (patients, interventions, comparison, outcomes) to formulate the review questions. Three organizations advise the consideration of additional aspects, for example, the context and the setting (7–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Two organizations (7;Reference Haas, Breyer, Knaller and Weigl8) recommend preliminary searches to support the planning process. The order of the steps (scope, conceptual framework, review question, preliminary searches) varies between the organizations. The Cochrane Collaboration and the CRD emphasize the importance of perspective (narrow versus broad) for the scope to decide whether to lump or split the review question.
Study Designs to Include
A specification of relevant study designs is given by three organizations (Reference Armstrong and Waters6–Reference Haas, Breyer, Knaller and Weigl8). It is emphasized that a wide range of study designs guided by the research question should be included, rather than the classic hierarchy of evidence, due to the broader scope and diversity of interventions as well as methods. All organizations agree that RCTs, cluster-RCTs, controlled before-and-after studies (CBA), or before–after studies and interrupted time series (ITS) should be considered for inclusion (Reference Armstrong and Waters6–Reference Haas, Breyer, Knaller and Weigl8). Various other quantitative study types (e.g., non-RCTs, historically controlled studies), qualitative studies (e.g., focus groups, interviews), and economic evaluations are also quoted.
Searching for Literature
All organizations recommend searching a wide range of databases relevant for the addressed topic (e.g., education, transport, engineering) in addition to medical databases (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The Cochrane Collaboration, CRD, and NICE recommend the use of free text words due to the heterogeneous terminology and poor indexing of public health concepts (Reference Armstrong and Waters6;7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). All organizations recommend additional searches (e.g., hand searches, contacting experts, organizational Web sites).
Quality Assessment
Guidance on the quality assessment of included studies (quality assessment of the evaluation of the intervention [study quality]) is given by all organizations (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The Cochrane Collaboration recommends the use of the “Quality Assessment Tool for Quantitative Studies” for RCTs and all nonrandomized study designs (Reference Armstrong and Waters6). None of the organizations suggest any tool for uncontrolled studies, while prima facie criteria are suggested for qualitative studies. GOEG refers to its own checklist for systematic reviews and cohort studies and also suggests criteria for qualitative studies. They do not provide recommendations for any other study design (Reference Haas, Breyer, Knaller and Weigl8). CRD discusses aspects of risk of bias for different study designs but only presents quality criteria for RCTs (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). NICE uses its own checklists to assess the internal validity of quantitative as well as qualitative studies (7).
Data Extraction
CRD, NICE, and GOEG address data extraction in separate sections (7–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). All three organizations recommend extracting data according to the PICO scheme and setting. CRD and NICE recommend extracting data on context, the integrity of the intervention, and the theoretical underpinning of the primary studies (7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). NICE recommends the use of a combination of narrative summaries and evidence tables (7). CRD endorsed the development of data extraction forms according to the review question (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The Cochrane Collaboration does not give explicit recommendations for data extraction in a separate section but mentions data extraction checklists and the importance of extracting data on PHI integrity (Reference Armstrong and Waters6).
Theoretical Framework
The organizations use theoretical frameworks in different ways. NICE uses a theoretical framework to identify relevant research questions (7). CRD proposes the use of a theoretical framework to group the interventions or include interventions that are based on one or more specific theories (inclusion criteria) (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The theoretical framework of the primary studies is part of the quality assessment (quality of the intervention). The Cochrane Collaboration suggests grouping the interventions according to their theoretical basis for tabulation, or for narrative or quantitative synthesis. GOEG applies the theoretical framework within the data synthesis to identify the components that determine effectiveness (Reference Haas, Breyer, Knaller and Weigl8).
Integrity of Intervention
Integrity is addressed by three organizations (Reference Armstrong and Waters6;Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Integrity of the primary studies is either considered as part of the quality assessment of the intervention or data extraction (description of intervention). The Cochrane Collaboration recommends collecting (description of studies), analyzing (e.g., factors influencing the effectiveness), and synthesizing data based on integrity (Reference Armstrong and Waters6).
Heterogeneity
All of the organizations emphasize the importance of taking heterogeneity into account while reviewing PHIs. They describe the various sources of heterogeneity (e.g., methodological, statistical, PICO, setting, context). The most common suggestion for dealing with heterogeneity is to perform subgroup analysis (7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). NICE recommends using a random-effects model for meta-analyses and performing a meta-regression if necessary (7). CRD suggests using Harvest plots.
Integrating Qualitative and Quantitative Studies
GOEG and the Cochrane Collaboration report on methods to integrate qualitative and quantitative evidence (Reference Armstrong and Waters6;Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). GOEG refers to a mixed methods approach, but without further description (Reference Haas, Breyer, Knaller and Weigl8). Moreover, the synthesis methods, realist reviews (Reference Armstrong and Waters6) and interactive domain model suggested by GOEG (Reference Haas, Breyer, Knaller and Weigl8) can be considered as holistic methods to combine qualitative and quantitative studies. The Cochrane Collaboration recommends three different approaches (Reference Armstrong and Waters6): The multi-layered model of social determinates in health, including three types of syntheses in the same review (meta-analysis, qualitative synthesis of views, mixed methods), or narrative synthesis (Reference Armstrong and Waters6).
Ethics, Equity, and Inequality
All organizations recommend considering equity aspects in the assessment. The Cochrane Collaboration incorporates equity aspects in different review steps (locating studies, indicators for inequalities, subgroup analysis, applicability) (Reference Armstrong and Waters6). NICE requires addressing disadvantaged groups throughout the entire review process (7). GOEG mentions equity aspects in some areas of the publication, but there are no explicit recommendations on how to integrate these into the HTA. CRD considers equity aspects in relation to the outcome definition (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9).
Sustainability
Two organizations address the sustainability of PHIs (Reference Armstrong and Waters6;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). The Cochrane Collaboration recommends seeking data on outcome patterns as well as contextual and project factors to assess sustainability. CRD indicates that, if long-term outcomes of the review are defined, attention should be paid to the validity of interim or surrogate outcomes (Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Both organizations evaluate sustainability as part of the quality assessment of included studies (Reference Armstrong and Waters6;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9).
Context
All organizations recommend considering information on context (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). CRD, the Cochrane Collaboration, and GOEG emphasize that the context is an important aspect for the generalizability of the results. Furthermore, CRD and GOEG suggest analyzing the effect of the context on the effectiveness of the intervention (Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9).
Applicability, Transferability, External Validity, and Generalizability
Issues related to applicability are also addressed by all organizations (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). NICE assesses the external validity and applicability using their own checklists (7). The Cochrane Collaboration recommends a tool to assess the applicability and transferability (Reference Armstrong and Waters6;Reference Wang, Moss and Hiller11). GOEG and CRD do not distinguish between the four terms (Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Both organizations recommend applying a checklist to assess the applicability of PHIs to different contexts (Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9;Reference Wang, Moss and Hiller11).
DISCUSSION
All four HTA manuals for PHIs consider similar (additional) methodological aspects (e.g., assessment of context). The methodology of specific steps is basically similar. However, in detail, the recommendations often vary widely without an obvious justification. Unjustified heterogeneity concerns the quality assessment, assessment of applicability and integration of qualitative and quantitative evidence. Furthermore, detailed descriptions of most process steps (e.g., integrity, heterogeneity, sustainability, context, and applicability) are missing, although these descriptions are necessary to guide the preparation for less experienced reviewers. Heterogeneity and lack of comprehensiveness are probably caused by the diversity and complexity of PHIs, which makes a detailed description and standardization of methods suitable for all research questions almost impossible.
There is a tendency for HTAs of PHIs to have an extended scope compared with medical clinical interventions. Comprehensive groundwork before conducting literature synthesis is particularly important in the area of public health to account for the high complexity of most interventions and to optimally adjust the methodology to the research question (Reference Craig, Dieppe, Macintyre, Michie, Nazareth and Petticrew12). Therefore, most organizations suggest exploratory work to develop the scope and a conceptual framework. There are different approaches using the theoretical framework in the process of preparing the HTA. The guidance documents for all organizations recommend structuring certain steps of the process (research question, inclusion criteria, data preparation, data synthesis). This demonstrates the broad applicability of a conceptual framework through the entire process of preparation for complexity in general (Reference Anderson, Petticrew and Rehfuess13). Furthermore, the theoretical framework is important as a part of the background and discussion of findings (Reference Armstrong and Waters6).
All HTA methods are based on systematic reviews (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). None of the organizations focus only on RCTs (Reference Armstrong and Waters6–Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). There seems to be a consensus that it is unreasonable to rely merely on evidence from RCTs (Reference Armstrong and Waters6–Reference Haas, Breyer, Knaller and Weigl8). The included study designs are flexibly adapted to the research question to achieve the answer in the best possible way. On the one hand, this reflects the broad perspective and complexity of PHIs (reference). On the other hand, this approach takes into account that randomized designs are often not feasible or are unethical for the evaluation of public health interventions (Reference Osrin, Azad and Fernandez14;Reference West, Duan and Pequegnat15).
No standardized terminology has evolved to label a PHI. Index terms for public health concepts are poor or unavailable, and the bibliographic language is heterogeneous (Reference Armstrong and Waters6;7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9). Therefore, search strategies should cover a wide range of databases, text words should be used freely and additional searches are important. A further challenge is the use of study filters to restrict the study design because different study designs are considered, and study designs may be termed differently (e.g., difference in difference analysis versus controlled before-and-after study). Consequently, the decision between the sensitivity and the precision of the search strategy is especially important for PHIs (Reference Armstrong and Waters6;7;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9), and a more pragmatic and iterative approach may be used (7) to optimally balance this conflict.
Internal validity assessments of quantitative studies differ between the organizations. In particular, the risk of bias assessment of observational studies and qualitative studies varies between the different organizations (Reference Armstrong and Waters6;Reference Sanderson, Tatt and Higgins16).
All organizations agree that data extraction should include detailed information on PICO and setting. Additional aspects of data extraction, such as context, theoretical underpinning, and integrity of the intervention, vary between the manuals, and the necessary data for the individual components are not further specified. The heterogeneous and imprecise reporting might be due to the diversity and complexity of PHIs. Integrity and context should be assessed by all organizations. Statements related to the intervention integrity refer mainly to the assessment of included studies. Indeed, the Cochrane Collaboration recommends analyzing and synthesizing information related to intervention integrity and describes relevant aspects. However, it is not elucidated how the assessment and synthesis should be performed such as methods suggested for other complex interventions (Reference Pawson, Greenhalgh, Harvey and Walshe17).
Standard statistical methods are predominantly suggested to deal with heterogeneity. Only NICE suggests meta-regression, and CRD suggests Harvest plots to analyze the differential effects of the intervention (7;Reference Ogilvie, Fayter and Petticrew18). We found few suggestions on how to use the different methods and no information for what type of heterogeneity (content related, statistical, methodological) the individual method is suitable for, although detailed methodological literature from the area of complex interventions on analyzing heterogeneity quantitatively still exists (e.g., graphical methods or advanced meta-analysis methods) (Reference Anzures-Cabrera and Higgins19–Reference Mavridis and Salanti21).
GOEG and the Cochrane Collaboration address the integration of qualitative and quantitative evidence (Reference Armstrong and Waters6;Reference Haas, Breyer, Knaller and Weigl8). Both segregated and integrated designs are described. Segregated methods are characterized by a prior synthesis of qualitative and quantitative studies separately, and a subsequent synthesis of the two syntheses (Reference Sandelowski, Voils and Barroso22). Two different segregated methods are described in the Cochrane publication: narrative synthesis and combining the findings of meta-analysis with qualitative analysis of reviews. However, neither is described in detail. GOEG mentions mixed methods as a possibility. In integrated methods, the synthesis of qualitative and quantitative data is performed in parallel and integrated design (Reference Sandelowski, Voils and Barroso22). Three different integrated methods are suggested: the realist reviews (Reference Armstrong and Waters6), the interactive domain model (Reference Kahan and Goodstadt23) (GOEG), and the multi-layered model of social determinates (Cochrane Collaboration) (Reference Dahlgren and Whitehead24).
All organizations consider the assessment of equity/ethical aspects as important, but do not provide any detailed descriptions on evaluation methods and integration of results in the review. Recent recommendations on the consideration of equity aspects in systematic reviews entail a checklist to assess the applicability to disadvantaged population groups (Reference Tugwell, Petticrew and Kristjansson25–Reference Welch, Brand, Kristjansson, Smylie, Wells and Tugwell27). However, they were developed after the development of the method guides assessed in this study.
Assessing the sustainability of outcomes is important to evaluate the long-term effectiveness of interventions. Intervention characteristics, context, capacity, processes, and interactions can all influence the sustainability of the intervention outcome/effectiveness (Reference Wiltsey Stirman, Kimberly, Cook, Calloway, Castro and Charns28). The Cochrane Collaboration provides some support in assessing the sustainability by suggesting relevant factors and assessing the pattern of outcomes. Given the large number of factors that need to be measured and reported, it is difficult to assess the sustainability of primary studies. However, one way to evaluate the sustainability is to assess the trend of outcomes after implementation of the included studies and consider the body of evidence. Another way would be the use of surrogate or intermediated outcomes because their effect on long-term outcomes has been validated.
Information on context can be used for two purposes. On the one hand, context can feed the analysis of barriers and facilitators to the intervention. On the other hand, it can be applied for the assessment of the applicability to the context (e.g., the country in which the PHI should be implemented). The organizations describe the context primarily as a part of the assessment of applicability. The approaches to assess the applicability of the included studies vary. The main reasons include the differing definitions of applicability, transferability, and external validity (generalizability); furthermore, some organizations use these terms interchangeably (Reference Haas, Breyer, Knaller and Weigl8;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9;Reference Burchett, Umoquit and Dobrow29). Most recommend checklists to assess the applicability of the intervention. Transferability and applicability of a complex intervention can only be assessed in relation to a particular population and context (Reference Burford, Lewin, Welch, Rehfuess and Waters30).
Therefore, one would expect that the organizations with international target audiences to focus on the external validity because an analysis of the applicability for all imaginable patients and contexts is not possible. Consequently, an assessment of applicability seems to be feasible only if it is performed by the end-user of the HTA. The initial provision of a detailed description of applicability information and subsequent performance of an example assessment of applicability for one target population and context that guides the assessments of the end-users may serve this purpose (Reference Burford, Lewin, Welch, Rehfuess and Waters30). An applicability assessment of the individual included studies can be supplemented by assessing the external validity of the body of evidence, i.e., the comparison of findings across studies.
The U.S. Preventive Services Task Force (USPSTF) published a manual after our systematic literature search (31). In contrast to the included manuals, this focuses on all (individual and population level) primary and secondary preventive services and not only on PHIs. As expected, the methodology is more similar to the assessment of medical interventions (e.g., focus on RCTs) than to the assessment of PHIs as described in the included manuals. Thus, only a few suggestions dealing with the complexity of PHIs can be found in the USPSTF manual. However, it is noteworthy that the USPSTF uses modelling for linking evidence (e.g., of a diagnosis-therapy-chain).
Our study has some limitations. First, we searched only for literature specifically related to HTAs of PHIs. Literature on other complex interventions (e.g., psychological interventions) and literature on complex interventions themselves were not included. Second, we included only manuals in German, English, Spanish, and Italian. Third, in the CRD guidance, the PHIs are only one section of a larger manual. Additionally, the Cochrane manual must be considered in connection with the Cochrane Handbook of Systematic Reviews (section 3–11) (Reference Armstrong and Waters6;Reference Akers, Aguiar-Ibáñez and Baba-Akbari9;Reference Higgins and Green32). It can be assumed that aspects that are not addressed in the public health specific documents should be performed as described in the “higher level” manuals of the respective institutions (e.g., Cochrane Handbook) (Reference Higgins and Green32).
CONCLUSION
The approaches used for HTAs of PHIs seem to be generally broader and more flexible than those for clinical interventions, which can be justified by the high complexity of PHIs. There is a tendency to identify the intervention components and other factors that influence the effectiveness and transferability of interventions/HTAs (complex perspective on a complex intervention) rather than to assess the effectiveness of an intervention in a more general way (simple perspective on a complex intervention) (Reference Petticrew, Anderson and Elder33).
Research is needed to further develop methods for HTAs of PHIs and ascertain the advantages and disadvantages of different approaches (e.g., realist synthesis versus systematic review), taking into account the research questions (e.g., broad versus focused perspective). This would positively contribute to the harmonization of the methods and consequently the usability for end-users (e.g., decision makers) of HTAs of PHIs. The assessment of quality, integrity, sustainability, equity aspects, context, and applicability require detailed reporting, especially if the influence of individual factors is to be quantified. A premise for accomplishing this is the improvement of reporting of primary studies.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S0266462317000228
CONFLICTS OF INTEREST
The authors have nothing to declare.
