Although there is no univocal agreement on what defines a complex health intervention, most definitions share some common features. Aspect of complex interventions often highlighted includes flexibility, nonstandardization (the form depend on the context), multiple interacting components, and nonlinear causal pathways (Reference Petticrew1). Health technology assessment (HTA) of complex health interventions, such as disease management programs, lifestyle interventions, and digital health interventions, may be challenging due to, for example, heterogeneous study designs and outcome reporting (Reference Petticrew1–Reference Petticrew, Anderson and Elder3). The European Union (EU)-funded project INTEGRATE-HTA offers guidance addressing specific challenges of complexity in a series of methodological documents (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4–Reference Booth, Noyes and Flemming8) integrated within the overarching INTEGRATE-HTA Model (Reference Wahlster, Brereton and Burns9). This study presents key messages from the Guidance for Assessing Effectiveness, Economic Aspects, Ethical Aspects, Socio-cultural Aspects and Legal Aspects in Complex Technologies (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4).
The INTEGRATE-HTA Model (Reference Wahlster, Brereton and Burns9) enables a coordinated assessment of these assessment aspects along with context and implementation issues (Reference Pfadenhauer, Rohwer and Burns6) and patient characteristics (Reference van Hoorn, Tummers, Kievit and van der Wilt5). A logic model (Reference Rohwer, Booth and Pfadenhauer7) provides a structured overview of the factors and aspects surrounding the technology and visualizes the assessment results. The understanding of integrated HTA that underpins INTEGRATE-HTA does not only include considering multiple assessment aspects individually, but also how these aspects are related, and how these interrelationships affect the assessment process and outcome.
INTEGRATE-HTA identifies five key aspects of complexity relevant to HTA; multiple and changing perspectives, indeterminate phenomena, uncertain causality, unpredictable outcomes and historicity, time and path dependency/ethical complexity (Reference Lysdahl and Hofmann10). These characteristics can be illustrated by the example of home based palliative care, where the interaction between the many different stakeholders entail multiple and changing perspectives, the range of models and different target populations means that the technology cannot be strictly defined, the individualized care and flexibility of services entail uncertain causal pathways between intervention and outcome, and the outcomes themselves may be unexpected due to many uncertainties involved (Reference Brereton, Wahlster and Lysdahl11). As all technologies are to a certain degree complex, the guidance recommends to make a systematic consideration of whether complexity is relevant for a given assessment, and provides structured help for doing so.
The guidance builds on previous efforts, including the HTA Core Model (12) and the Health Technology Assessment Handbook (Reference Kristensen and Sigmund13). Thus, it serves to complement and expand upon existing HTA guidance. The guidance is based on an understanding that all five aspects must be addressed when complex technologies are to be assessed including ethical, socio-cultural, and legal aspects, which still tend to be ignored in current HTA practice. This comprehensiveness is crucial as a complex health technology mutually interacts with the societal context. This calls for sensitivity toward ethical, socio-cultural, and legal norms to understand the impact and acceptance of the technology. The objective of this article is to give an outline of this Guidance and its elements, as well as to indicate the added value of applying it in HTA, in an integrated manner.
METHODS
The guidance comprises five assessment dimensions in HTA; effectiveness, economics, ethics, socio-cultural, and legal aspects. According to the nature of the respective fields the methods by which the guidance was developed, and its focus, differs across the five dimensions. For example, legal research and practice has developed from a different paradigm than assessment of clinical intervention effectiveness. However, one common feature is that the specific guidance parts draw on existing and published knowledge in the fields of HTA, complexity science, as well as knowledge of research methodologies in the five respective fields. Another common feature is that all guidance parts were informed by stakeholder involvement, including healthcare professionals, patients, and relatives from six European countries in the scoping process, as well as internal and external HTA researchers.
The concepts and methods suggested in the different guidance parts have been applied in a case study on reinforced home based palliative care (Reference Brereton, Wahlster and Lysdahl11), and subsequently adapted through collaboration and feedback between the researchers in the project. The INTEGRATE-HTA working group consist of HTA researchers, and experts in all relevant aspects of HTA, from five European countries. Finally, the guidance was revised after valuable feedback from internal and external reviewers. Throughout the process of guidance development, there have been extensive collaborations aiming at harmonization and integration across various parts of the guidance.
RESULTS
An overview of each of the assessment aspects of the INTEGRATE-HTA guidance (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4) is presented in the following, that is, what content and tools the user will find when applying the guidance, and the expected benefits thereof. Complexity exists along a spectrum as there is no clear-cut distinction between simple and complex technologies. This makes the systematic approaches to describe aspects of complexity presented in this guidance important to understand when the complexity matters for HTA and for decision makers. This is relevant for all assessment aspects and for most health technologies.
Guidance for the Assessment of Effectiveness
An assessment of effectiveness comprises a variety of steps, from forming the question, to designing and conducting searches, identifying evidence, appraising the quality of and synthesizing evidence, as well as effectively communicating the results. Comprehensive resources outlining this process exist (12;Reference Higgins and Green14), thus for this guidance we focus on two aspects of the effectiveness assessment process which may prove especially problematic for complex interventions. Specifically, the effectiveness guidance offers a flexible guide to help users choose appropriate methods for dealing with heterogeneous study designs and for synthesizing effectiveness evidence. It is based on the assumption that there is no one-size-fits-all solution for dealing with heterogeneous study designs and synthesizing evidence.
As such, for any assessment, multiple options may be suitable and appropriate, including a range of quantitative, qualitative, and mixed-methods approaches. Nevertheless, case-specific aspects can make certain methods more suitable and appropriate than others. In this part of the guidance, we considered these case-specific aspects to be the specific research question; the specific technology under assessment and the system in which it exists; the resulting complexity; and the available evidence. This guidance describes these issues in detail and discusses their potential implications for the choice of method.
The application of this guidance involves working through a series of steps. The first three steps occur a priori before beginning the assessment, and include (i) conducting a comprehensive scope of the effectiveness assessment; (ii) gaining a thorough understanding of the characteristics of available methods; (iii) (conditionally) specifying methods a priori. For some technologies, these steps may be sufficient for choosing appropriate methods for dealing with heterogeneous study designs and synthesizing evidence. For others, however, it may still be unclear whether these methods are appropriate, and thus the a priori decision should be treated as conditional.
Once the assessment commences and the potentially relevant studies have been identified, the final two steps may be necessary, including (iv) assessing methodological and clinical heterogeneity in the identified evidence base; and (v) specifying final decision on methods.
In working through these steps the user must consider issues such as the type of question being asked (e.g., “Is the technology effective?” as compared to “What parts of the intervention are most effective?”), as well as the specific definitions for the population, technology, comparison and outcomes to be included. This may sound straightforward, but in the presence of complexity the hammering down of these issues may become substantially more difficult, as the boundaries around the technology begin to blur, as the number of outcomes and stakeholders increases, and as context and implementation become more important. Regarding dealing with a heterogeneous evidence base, the guidance provides information to help the user to consider and balance directness of evidence with the risk of bias, so that the best available and most informative evidence can be used, while the risk of bias within the evidence base is explicit, understood, and dealt with appropriately.
The inclusion of only randomized controlled trials, for example, will ensure that the risk of bias of included studies remains relatively low, compared with the inclusion of other nonrandomized study designs such as interrupted time-series or controlled before–after studies. In some cases, however, these nonrandomized study designs can provide much more direct evidence than highly controlled randomized controlled trials, and may thus be considered more informative. This guidance document describes a range of quantitative meta-analytical and summary methods for synthesizing evidence of effectiveness, and discusses implications for the various facets of complexity.
The following simple examples should help the reader understand how certain facets of complexity may match well to specific evidence synthesis methods: (i) If the assessed technology is not a single technology, but rather several variations of a technology, network meta-analysis may allow the assessment to compare variations of the technology that have not been compared directly in the primary literature. (ii) If varying contextual aspects are likely to influence effectiveness, meta-regression could be useful for assessing the effect of such aspects. (iii) If included studies are simply too heterogeneous with regard to the population, technology, outcomes, or study methods to justify statistical pooling, a graphical summary approach like the forest plot without a summary statistic or the harvest plot can summarize study effects in a clear, concise manner.
Additionally, the guidance briefly introduces how stakeholder input, as well as qualitative methods and mixed-methods may also play an important role. One limitation of this guidance is that it does not provide concrete guidance for assessing the risk of bias from heterogeneous types of evidence. Currently, there is no widely accepted standard for assessing risk of bias across study designs. Much work, however, is happening in this field, and some resources do exist for assessing risk of bias across study designs (Reference Wells, Shea and O'Connell15;Reference Sterne, Hernan and Reeves16).
Each effectiveness assessment is at least somewhat unique, meaning there is a limit to how specific such guidance can be. In the effectiveness guidance; however, a range of options for study design inclusion and evidence synthesis are documented and described, and it is emphasized that the user makes decisions regarding these methods only after substantial consideration of the research question, the technology and the system in which it exists, the resulting complexity and the existing evidence. The guidance also suggests that controlled flexibility in deciding upon methods may be necessary and beneficial to ensure that effectiveness assessments provide the best possible evidence for informing decisions.
Guidance for the Assessment of Economic Aspects
Complex interventions and particularly those that interact with context and setting throw up special problems for health economic assessment (Reference Shiell, Hawe and Gold17). We developed guidance that focuses on the use of a systems approach for undertaking model based economic evaluation of complex interventions in a complex setting (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4). The guidance on practice is based on a combination of problem structuring methods and quantitative modeling. Although conceptual frameworks exist for structuring the consideration of public health interventions (Reference Squires, Chilcott, Akehurst, Burr and Kelly18), no similar conceptual frameworks exist for more generic complex interventions.
A systems approach to economic evaluation is recommended to provide a useful conceptual framework for addressing several of the issues raised by complexity. It takes as its starting point Step 1, “Definition of the HTA objective and technology” and Step 2 “Creation of an initial logic model to define evidence needs” of the INTEGRATE-HTA Model (Reference Wahlster, Brereton and Burns9). The economic modeling guidance expands on the HTA logic modeling by developing descriptions of the health systems that enable the potential impact of the intervention on economically relevant, resource, cost and effectiveness outcomes to be made explicit. The systems approach enables the economic conceptual modeling to be made coherent with the other HTA dimensions, including effectiveness, ethical, socio-cultural and context, and legal elements (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4).
The guidance recommends including a formal consideration of aspects of complexity in developing an understanding of the decision problem and scope for economic evaluation and updating this during the project. This should include in-depth exploration of the economic implications of those aspects of complexity described in the introduction, including indeterminate phenomena (e.g., understanding of variation in the intervention, its implementation, and in the system into which it will be introduced), uncertain causality (e.g., factors that influence how the intervention impacts on the system and economic outcomes) and historicity, that is the time, place, and system context of an intervention.
The guidance also recommends an increased role for multi-agency cost consequence analysis (CCA) to support decision making in the presence of multiple perspectives. The use of formal consideration of complexity aspects to ensure that stakeholders have a comprehensive understanding of the evidence base and the economic model, and specifically their limits of applicability to ensure appropriate interpretation of quantitative economic outcomes in decision making.
Recommendations for research include methodological development on the role of computational complexity science methods to support health economics within HTA, the use of computational modeling techniques, such as agent based modeling and social network analysis for understanding the health economic impact of adaptive behavior and co-evolutions of intervention and setting within HTA and the modeling of behavior within health economic models. Methods for measuring, valuing and incorporating non-health benefits into the HTA process require development.
Guidance for the Assessment of Ethical Aspects
In the same manner as with the guidance for assessing effectiveness and economic aspects in HTA the guidance on ethical aspects is developed to address methodological challenges posed by the complexity of the intervention. The point of departure for the ethics guidance is the fact that several ethical approaches are available for use in HTA, for example, principlism, casuistry, wide reflective equilibrium, social shaping of technology, interactive technology assessment, the triangular model, the HTA core model, and the socratic approach. These approaches may be more or less suitable, depending on the complexity of the technology and the context. For instance, the ability to detect ethical issues embedded in the many/various/unpredictable outcomes may vary between the approaches. This calls for a methodical flexibility, which may be demanding for the assessor. Hence, the guidance presents a procedural framework to assists in how to select, modify/expand, and use the various existing ethical approaches.
To deal with ethical issues in a structured way the guidance suggests a stepwise application. The first two steps address the selection of the most appropriate ethical approach. This starts with assessing the complexity of the technology against a set of ethically relevant complexity characteristics, that is. determining the technology's “complexity profile” (Step 1), followed by assessing how the various ethical approaches fit this complexity profile in general (Step 2). The choice of ethical approach also includes assessing the ethical approaches against some relevant local HTA characteristics (i.e., the integration perspective, and the aim for assessment versus appraisal).
Despite a systematic and thorough selection process, the standard ethical approach may not fit perfectly for the HTA in question. For example, the best available approach may not address all important ethical issues of the technology at hand. Accordingly, it may be necessary to modify the approach, which is carried out in Step 3. As detailed descriptions of how to apply the different ethical approaches are available in the literature, the approaches are only introduced in the guidance and references to further reading is provided to assist (nonethicists) in the application process (Step 4). Furthermore, the need for bilateral integration across HTA aspect in the application process is highlighted. The outcomes of the ethical assessment (Step 5) are not constrained to specific ethical areas (e.g., concerning the patient group, the health technology, the implementation, or the HTA process), as the outcomes depends on the chosen approach and how it is applied for the health technology at hand.
When following the steps of the guidance the users are provided with tools facilitating the choice and adjustments of the ethical approaches, such as illustrating the meaning and implications of the five complexity characteristics, providing the overview of available ethical approaches, and listing some of relevant features of each approach. Moreover, the guidance indicates the value of stakeholder involvement in the different steps, for example, as a source of information when determining the complexity profile of the health technology (Step 1) and in validating the outcome of the ethical assessment (Step 5). Overall, the guidance increases the awareness of challenges and opportunities embedded in the ethical approaches and of applying them in a local context, that is, as other parts of the guidance, the ethics part is dynamic and adaptive to the complexity of the technology and the context rather than offering a one-size-fits-all approach.
Guidance for the Assessment of Socio-cultural Aspects
This part of the guidance provides a structured approach to include the assessment of socio-cultural aspects in HTA, and to increase the user's understanding of the assessment of socio-cultural aspects of health technologies, methodological challenges and solutions involved. Socio-cultural aspects are of special interest in complex interventions due to potential interactions between the technology and the socio-cultural context in which the technology would be used.
The guidance on socio-cultural aspects offers new methodological tools and procedures for the assessment of socio-cultural aspects. It provides an inductively developed framework and a stepwise assessment process as well as an option to identify and address cultural heterogeneity of different social groups. The assessment process contains five-steps: Step 1, the assessment of the complexity of a technology; Step 2, the identification and prioritization of relevant socio-cultural aspects and stakeholders; Step 3, the validation of identified aspects; Step 4, the assessment of the prioritized aspects; and Step 5, the presentation of the evidence.
The socio-cultural framework is a comprehensive tool to identify and evaluate socio-cultural aspects on different levels of social organization and from the perspectives of social and cultural groups. It can be applied on each of the five steps in the assessment process. For example, in Step 1, to identify relevant complexity issues from a socio-cultural perspective, based on the list of complexity characteristics provided in the ethics part of the guidance. The framework consists of main categories and subcategories and outline aspects of interest for the socio-cultural assessment within these categories. The first category is “the social understanding of the health issue” and the second is “the social understanding of the technology.” The latter covers “the perceived usefulness,” “the knowledge and understanding of the technology,” “attitudes to and the acceptance of the technology,” and “the risk perception and handling.” The last main category is “the socio-cultural aspects of the technology's implementation,” which covers “socio-cultural aspects of the target groups,” “social inequality,” “the user-professional-relationship,” and “relationships between professionals providing the technology.”
When applying this framework the user will find support in terms of thorough description of “the aspects of interest” and lists of related assessment questions (e.g., “How do different professional cultures influence the provision of the technology?”, “Do different groups perceive risks related to the technology differently?”, “Why does the technology work/is accepted in one cultural context and not in another?”, “Does the implementation/use of a technology lead to social inequalities?”), which facilitates a systematic assessment of the different aspects. Additionally, illustrating examples from application in home based palliative care are provided.
Application of the guidance facilitates the explicit identification and evaluation of socio-cultural aspects and thereby generates a broader understanding of these aspects from different perspectives as well as at different levels (micro-, meso-, and macro-level) of the social organisation. It also takes into account linkages to context and implementation and offers an option to include and systematically address cultural heterogeneity of perspectives within the HTA process.
Guidance for the Assessment of Legal Aspects
Legal aspects of health technologies are diverse and difficult to assess within a generic framework. This problem is exacerbated for complex technologies and interventions where the range of potentially relevant legal aspects is more diverse. An example might be a drug-supported psychological intervention in which for some parts of the intervention the legal issue of market authorization is most crucial (the pharmaceutical product) while others rather have implications for privacy and data protection (the psychological interventions). Furthermore, the importance of different legal aspects is dependent on the addressee and user of the HTA (i.e., the level of the decision that shall be based on the HTA-report). The legal aspect of informed consent might not be important for the hospital management users of an HTA-report when deciding to buy a new medical device, but used as a guideline for the clinical staff the HTA-report must point out that informed consent of the patient to be treated with the same device is an absolute condition. Assessing legal aspects in HTA is, therefore, heavily dependent on the identification of the most important legal implications, depending of the respective technology assessed as well as the relevant decision level on which the HTA-report is to be used.
The INTEGRATE-HTA guidance for legal aspects in HTA aims at supporting HTA researchers in identifying the respective relevant legal fields and the potential need for further legal inquiry. The guidance focuses on nine legal core issues, which have been identified as mostly crucial for the assessment of different technologies and are related to different decision levels. Three of these core issues, the informed consent, alternative forms of consent and privacy and data protection can be subsumed under the term “Autonomy of the Patient,” two concern the legal field of market authorisation, specifically of medical devices and medicinal products. The other four legal issues are clinical trials, intellectual properties, reimbursement in public healthcare systems and special medical fields. For each of these issues the guidance provides a brief introduction followed by: (i) an initial question; (ii) a short explanation of the legal issue, often including examples; (iii) an overview over relevant legal sources; (iv) relations of the legal aspect to other parts of the HTA; and finally (v) a reference on the decision level on which the aspect is relevant.
The guidance is applied by answering the initial question, using the further explanations to assess whether or not the legal aspect is of relevance for the respective technology. If the aspect is of importance, the further steps can be used to assess the need for and to start an in-depth analysis of the aspect within the HTA. It is important to note that this generic guidance does not seek to substitute for case based legal advice, in assessments where the legal aspect is considered to be of paramount importance and potentially unclear, a professional legal advisor should always be consulted. In this way, the legal guidance provides a decent framework, not for a conclusive clarification of all possibly affected legal aspects of a technology but for identifying such aspects and for identifying when further measures are advisable or necessary.
DISCUSSION
The brief presentation of the five aspects included in the Guidance for Assessing Effectiveness, Economic Aspects, Ethical Aspects, Socio-cultural Aspects, and Legal Aspects in Complex Technologies, (Reference Lysdahl, Mozygemba, Burns, Brönneke, Chilcott and Hofmann4) reveals similarities but also some differences between them. Differences stem from disparities in the perspective, traditions and state of methodological development of the five disciplines. For instance, whereas there is a wealth of methodologies for assessing effectiveness, economic and ethical aspects, there is substantially less methodological guidance available for the assessment of socio-cultural and legal aspects. Hence, the latter two focus on developing new methods, while the former three focuses more on complementing and expanding on existing methods.
As the various parts of guidance are developed in the same vein, all parts of the guidance offers systematic, stepwise approaches, which also includes involving stakeholders and their perspectives systematically. Such a systematic assessment process is highly welcome in the “unclear terrain” of assessing complex health technologies. Additionally, all guidance parts emphasize the possible need for flexibility in choosing and applying methods when dealing with complex interventions. This flexibility reflects that the presence of complexity have stronger implications for the planning, conduct and interpretation of the HTA (which is the focus of the guidance) for some health technologies than for others. Hence, an understanding of the impact of technologies complexity is always useful to make appropriate methodological choices. A further common feature is that all parts of the guidance provide examples of how they can be applied, mostly drawing on the case study of home based palliative care (Reference Brereton, Wahlster and Lysdahl11). The value of providing examples may be particularly high when dealing with the challenges of complexity.
Throughout the guidance we have indicated the interrelations between the five assessment aspects and the possibilities for integration. For instance, the economic analysis may point out a need for addressing the ethical question about fair distribution of resources, which may relate to legal regulation of priority setting. Furthermore, we have explained how and where this guidance fits into The INTEGRATE-HTA Model (Reference Wahlster, Brereton and Burns9). This does not exclude the users possibility to apply one/some parts of the guidance separately (e.g., to assess only the legal issues of the technology). However, it is strongly recommended to address the different parts in a comprehensive and integrated HTA.
Nonetheless, the issue of integration across assessment aspects seems to be persistent in HTA (Reference Hofmann, Oortwijn and Bakke Lysdahl19), and we recognize that the methods for integration across assessment aspects need further development. Overlaps between assessment aspects are particularly relevant for ethical and socio-cultural aspects, explaining why they can be addressed together (e.g., in a common literature search) (Reference Lehoux and Williams-Jones20). The inherent links between the economics and the effectiveness assessment means that a close sharing of identified primary evidence, of extracted results, and of synthesized evidence is beneficial in assessing both assessment aspects. Moreover, in the application of the guidance in the case study of home based palliative care, we found several concepts and issues (e.g., access and availability) relevant across all assessment aspects, and that the perspective of one aspect can have impact on the assessment of other aspects.
CONCLUSION
The Guidance for Assessing Effectiveness, Economic Aspects, Ethical Aspects, Socio-cultural Aspects and Legal Aspects in Complex Technologies (Reference Pfadenhauer, Rohwer and Burns6) adds value to HTA by providing a practical step by step guidance on assessing complex health technologies. While it has been recognized that all technologies are to some degree complex, the key issue is to understand when the complexity matters for HTA and for decision makers, and specifically when special methods need to be used in the assessment of a technology. By putting forward systematic approaches to describe aspects of complexity this guidance seeks to enable appropriate methodological choices to be made. It brings together assessments from distinct disciplines, such as epidemiology, economics, ethics, law, and social theory. As this integration is not necessarily specific to the level of complexity of an intervention or system, the guidance can have a much broader potential relevance.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
