Health technology assessment (HTA), according to the European network for Health Technology Assessment (EUnetHTA), is a “. . .multidisciplinary process that summarizes information about the medical, social, economic and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, robust manner. Its aim is to inform the formulation of safe, effective, health policies that are patient-focused and seek to achieve best value” (1). Despite significant advances in recent years there are still major methodological gaps when complex technologies are to be comprehensively assessed. According to the UK Medical Research Council (MRC) complex technologies or complex interventions are characterized by several interacting components, the number and difficulty of behaviors required by those delivering or receiving the intervention, multiple groups or organizational levels targeted, many and variable outcomes, and explicitly permitted flexibility or tailoring of the intervention (2).
Following this definition, palliative care is an example of a complex technology. Palliative care can be implemented in different settings such as at home or in a hospice. It can be delivered by a nurse, a doctor, or an informal caregiver (such as a relative, friend, or neighbor). The provision of palliative care depends on the country-specific healthcare system, the degree of professionalization of services, and the geographical context. Patient characteristics and preferences can be diverse with respect to early or late stage terminal diseases, the experience of pain or emotional stress and despair, and the availability of family or a social network. The variety of relevant aspects such as effectiveness, economic, socio-cultural, legal, and ethical issues translates into multiple outcome parameters to be assessed for patients and their informal carers. These include quality of life or spiritual improvements for patients as well as the impact on caregivers. HTA needs to consider all of these aspects to provide conclusions that are meaningful for decision-making.
Current HTA is not well-equipped to assess complex technologies, as insufficient attention is being paid to the diversity in patient characteristics and preferences, context, and implementation issues and several other aspects, such as ethical, legal, and socio-cultural considerations. Notably, consideration of health system aspects such as “organizational and supportive systems within which all health technologies are delivered” have recently been identified as key challenges in HTA (Reference Horton3).
Furthermore, strategies to integrate the different aspects that determine the value of a technology in a coherent assessment are lacking in HTA. Currently, different aspects are usually assessed independently from each other and presented side-by-side. Therefore, decision makers tend to be tasked with integrating and interpreting the different results of a HTA in an overall conclusion. This important part of the process is necessarily based on the experiences and values of the decision makers, which are usually not been made transparent.
One approach to deal with this challenge is Multi-criteria Decision Analysis (MCDA). MCDA offers “a collection of formal approaches (ranging from qualitative to fully quantitative) to use multiple criteria in helping individuals or groups explore decisions that matter” (Reference Benton and Stewart4). One of these approaches, the EVIdence based DEcision Making (EVIDEM) framework was specifically developed for decision-making based on HTA. The EVIDEM framework consists of fifteen quantifiable core criteria (e.g., severity of disease, budget impact, validity of evidence), which are scored and weighted. Based on these weights and scores, an MCDA estimate is calculated. Qualitative considerations, such as ethical issues, can be taken into account using a contextual tool (Reference Wahlster, Goetghebeur and Kriza5;Reference Wahlster, Goetghebeur, Schaller, Kriza, Kolominsky-Rabas and EMN’6). EVIDEM has become a very sophisticated method to support decision-making. Still, it does not cover all relevant issues, such as context and implementation. Also, EVIDEM does not account for interactions between the assessment processes for different criteria, even though many criteria are interrelated, for example, the influence of context variables on effectiveness.
The aim of the European Union-funded INTEGRATE-HTA project was to develop concepts and methods for a comprehensive, patient-centered, and integrated assessment of health technologies that includes and considers effectiveness and economic, sociocultural, ethical, and legal issues, patient preferences and patient-specific moderators of treatment, as well as context and implementation issues. The methods for the assessment of each of these aspects were tested and demonstrated in a case study on palliative care (Reference Brereton, Wahlster and Lysdahl7). In doing so, distinct pieces of methodological guidance were developed (Reference Lysdahl, Mozygemba and Burns8 – Reference Van Hoorn, Tummers, Kievit and Van der Wilt12). The INTEGRATE-HTA Model describes the overall process of a patient-centered, comprehensive, and integrated HTA. In this article, we present the development of the INTEGRATE-HTA Model and its application in a case study of palliative care (Reference Wahlster, Brereton and Burns13).
METHODS
A multi-method, four-stage approach guided the development of the INTEGRATE-HTA Model: (i) the definition of dimensions of information that need to be integrated in HTA; (ii) a literature review of existing methods to integrate these different dimensions; (iii) adjustment of concepts and methods for assessing different aspects of complex technologies in the frame of an integrated process; and (iv) the application of the INTEGRATE-HTA Model in the case study on palliative care and subsequent revision.
Defining Dimensions of Information That Are Relevant for an HTA-Based Judgement
Based on literature reviews and deliberation within the research team, we defined four dimensions of information that need to be integrated. The first dimension comprises the different assessment aspects (e.g., effectiveness, costs, ethical, legal, social, and cultural aspects). The second dimension embraces the factors that can have an influence on the aspects to be assessed: contextual factors, factors related to the implementation and patient characteristics. The third dimension refers to the uncertainty that is related to each assessment and that needs to be adequately considered and communicated. Importantly, the fourth dimension including values, preferences, and experiences of the HTA researchers and relevant stakeholders frames the other three dimensions (Reference Wahlster, Brereton and Burns13).
Literature Review of Existing Methods to Integrate the Different Dimensions
A systematic literature search was performed to obtain a comprehensive overview of integrative approaches that cover the different dimensions described above. Articles on integration methods published in medical (Web of Science, Medline, PsycINFO) and nonmedical databases (Econlit, ASSIA, International Bibliography of the Social Sciences, Sociological abstracts) between January 2004 and April 2014 were assessed for eligibility. Integration methods were defined as existing methodologies for integrating different dimensions of information. These methods were appraised for applicability to HTA and were included, if they are able to integrate at least two of the four dimensions of information (Reference Wahlster, Brereton and Burns13).
Adjusting Existing Concepts and Methods to Assess Complex Technologies
From our perspective, an integrated assessment needs to start from the beginning of the HTA. Existing concepts need to be adjusted to account for the different dimensions of information (see above). Methods to assess distinct aspects (e.g., effectiveness, ethical aspects) need to be aligned to account for the complexity of technologies. The process for adjustment varied depending on the particular method. These processes have been described elsewhere (Reference Lysdahl, Mozygemba and Burns8;Reference Wahlster, Brereton and Burns13).
Application of the INTEGRATE-HTA Model in a Case Study on Palliative Care
We applied the INTEGRATE-HTA Model to the assessment of different models of home-based palliative care in a case study (Reference Brereton, Wahlster and Lysdahl7). The model was iteratively revised through interactions with other methodological guidance developed in the INTEGRATE-HTA project, during practical application and following external peer-review.
RESULTS
The INTEGRATE-HTA Model, structured in five steps, is shown in Figure 1. After defining the HTA objective and the technology (step 1), a logic model is developed (step 2). The logic model provides a structured overview of the current conditions regarding the health technology, relevant assessment aspects, patients’ characteristics, context, and implementation. In step 3, evidence on effectiveness, economic, ethical, legal, and socio-cultural aspects is assessed. In step 4, a populated logic model is provided that is structured by the HTA objective from step 1 and the assessment results produced in step 3. Finally, step 5 establishes a direct link with decision-making by applying decision support tools. Importantly, all steps of the INTEGRATE-HTA Model involve stakeholder consultations to provide the opportunity for clinical experts, academics, patients, relatives (including informal caregivers), the public, or other relevant stakeholders to contribute suggestions and provide feedback to the HTA project team. In the following sections, the five steps of the INTEGRATE-HTA Model are described in detail.
Figure 1. The INTEGRATE-HTA Model for an integrated assessment of complex technologies (Reference Wahlster, Brereton and Burns13).
Step 1: Definition of the HTA Objective and Technology
Step 1 defines the technology under assessment and the objective of the HTA, including relevant issues and outcomes to be assessed. Two key activities inform this step: a literature review and stakeholder advisory panels (SAP). The term “panel” refers to the collective information provided by individuals or groups independent of their location, as patients and professionals cannot always attend face-to-face meetings.
Stakeholders are consulted at the outset of the HTA to identify key issues associated with the technology to be assessed (Reference Brereton, Ingleton and Gardiner14). For complex technologies, consisting of different components, it is often not self-evident, what exactly the technology under assessment is. A literature review and discussions with the stakeholders may be necessary to agree on the final, or at least a preliminary definition of the technology. The definition of the technology should identify the common generalizable features it has within similar technologies; it should also specify which components are fixed and which components are adaptable to different settings (Reference Squires, Valentine and Grimshaw15).
A well-defined HTA objective is important for a coordinated assessment of all relevant aspects of health technologies. Therefore, the HTA objective should be structured according to specific decision criteria. The decision criteria can be selected from the scoping literature review on the assessment theme, a generic set of criteria (such as EVIDEM [16] or the HTA Core Model [17]), or the criteria of existing appraisal committees (such as NICE or the Dutch healthcare authority). The definition of the decision criteria should be consistent with the values of stakeholders as well as the input from the theoretical, methodological, and empirical literature. In addition, the different perspectives of stakeholders translate into different outcome parameters to be assessed for the technology of interest. The output of step 1 is the definition of the HTA objective, including relevant issues, outcomes, and the technologies to be assessed (such as models of home-based palliative care).
Application of the Model in Practice: A Specific and Policy-Relevant HTA Research Question for Home-Based Palliative Care
In the INTEGRATE-HTA case study, stakeholder consultations identified support for informal caregivers providing home care as a key concern in palliative care. These consultations and a review of the literature identified models of home care with and without an additional component of caregiver support, respectively, known as reinforced and non-reinforced models as the technology and comparator to be assessed. As such, the objective of the case study was to compare reinforced models of palliative home care versus non-reinforced models of palliative home care. Both models of care allow patients to receive care primarily at home. Reinforced models of home care will additionally include an intentional and explicit attempt to support informal carers (by, e.g., psychological support, sitting service, structured training) in addition to the care given to patients. The SAPs contributed their perspective on the relevance of the decision criteria for palliative care. The criteria selected were defined according to the glossary of the International Network of Agencies for Health Technology Assessment (INAHTA) (18) and the Joanna Briggs Institute (19). Step 1 resulted in the identification of the following HTA research question:
“Are reinforced models of home-based palliative care . . .
-
• acceptable,
-
• feasible,
-
• appropriate,
-
• meaningful,
-
• effective,
-
• and cost-effective
. . . for providing patient-centered home-based palliative care [compared with usual home-based care models of palliative care] in adults (defined as those aged 18 years and above) and their families?” (Reference Brereton, Wahlster and Lysdahl7;Reference Brereton, Ingleton and Gardiner14).
Step 2: Creation of a Logic Model to Define Evidence Needs
In step 2, a logic model is created to provide a structured overview of the specific technology and the system in which it exists, including the relevant issues of interest, outcome parameters to be assessed, patient preferences and moderators, as well as context and implementation issues. Where no suitable logic model can be identified from the literature, a logic model template (Reference Rohwer, Booth and Pfadenhauer11;Reference Rohwer, Pfadenhauer and Burns20) provides a starting point and is completed with the knowledge about the technology that was obtained in step 1 (Reference Pfadenhauer, Rohwer and Burns9). Various approaches such as conceptualization, literature searches, and stakeholder involvement can be used to populate the various elements of the logic model. These can also be applied when identifying and assessing relevant patient preferences and moderators (Reference Van Hoorn, Tummers, Kievit and Van der Wilt12). The SAPs and the HTA researchers review evolving versions of the logic model and provide feedback on its plausibility. Indeed, the final version of the logic model might require several iterations, drawing on additional SAP consultations and (nonsystematic) literature reviews. At the end of step 2, the logic model visualizes the aspects relevant for the assessment of the technology and any interactions between them.
Application of the Model in Practice: A Comprehensive Logic Model for Home-based Palliative Care with or without Additional Caregiver Support
A logic model was developed for reinforced and non-reinforced models of home-based palliative care as the technology and comparator of choice. The information about patient preferences and moderators (such as patient preferences for place of death) as well as context (such as rural or urban area) and implementation issues (such as implementation by generalized palliative care team or specialist palliative care team) was assembled from the initial consultation with the SAPs in step 1 and a review of the literature. Figure 2 shows the logic model for reinforced and non-reinforced models of home-based palliative care as the output of step 2 (Reference Brereton, Wahlster and Lysdahl7).
Figure 2. Example: Logic model of reinforced and non-reinforced home-based palliative care (Reference Brereton, Wahlster and Lysdahl7).
Step 3: Evidence Assessment
In step 3, the evidence is collected and assessed. The logic model resulting from step 2 illustrates likely interactions and scenarios for specific patient groups and compositions of contextual factors that are relevant for the technology and situation under assessment. These interactions need to be taken into account when doing the assessment (e.g., patient preferences can inform the search strategy for safety and effectiveness outcome parameters, but also for ethical, legal, and socio-cultural aspects) (Reference Lysdahl, Mozygemba and Burns8). Continuous consideration of the interdependencies between the various assessment procedures is essential to avoid redundancies (such as overlaps between the socio-cultural and the ethical assessment, see example below) and to enable complementary insights. The outputs of step 3 are evidence reports and standardized evidence summaries for each assessment aspect. Whereas the full evidence reports can be stand-alone reports for each assessment aspect, the purpose of the standardized evidence summaries is to provide a transparent and operational overview serving as input for further processing to populate the logic model.
Application of the Model in Practice: Evidence on Effectiveness, Cost-Effectiveness, Socio-cultural, Ethical, and Legal Aspects of Home-Based Palliative Care
Separate assessments were conducted for effectiveness, cost-effectiveness, socio-cultural, ethical, and legal aspects. In doing so, multiple assessment aspects were concerned with the issue of autonomy and shared decision-making, providing a variety of potential insights into the findings. For instance, the effectiveness assessment showed that structured training of informal caregivers increased their quality of life but not their psychological health (Burns et al., submitted manuscript). A possible explanation for this finding indicated by the socio-cultural assessment was that caregivers worry about increasing responsibilities as a consequence of the training. From an ethical perspective, carer autonomy is challenged with regard to voluntariness of role acquisition and the implementation, use and withdrawal of the structured training. Informal caregivers may feel morally obliged to take on a caregiving role as a result of promises made to the patient (e.g., to fulfil wedding vows) or social pressure. In parallel, the legal assessment identified that conflicts of interest about patient and carer autonomy can arise between patients and family carers (Reference Brereton, Wahlster and Lysdahl7).
Step 4: Mapping of Evidence
In step 4, the assessment results of step 3 are presented using the logic model developed in step 2 as a structure. Compared with the logic model developed in step 2, the populated logic model in the step 4 includes the decision criteria of the HTA objective (step 1) and the assessment results (step 3). The assessment results from step 3 are assigned to the different decision criteria of the HTA objective and then entered into the logic model. A detailed description of how to construct this logic model is published elsewhere (Reference Wahlster, Brereton and Burns13). The populated logic model enables a comprehensive, transparent, and integrated graphical presentation of all assessment results.
Whereas the logic model in step 2 specifies which types of evidence are relevant, the populated logic model in step 4 maps the various findings of the different methodologies and the interactions between them. It also allows for the consideration of different scenarios depending on variation in context, implementation, and patient characteristics. For instance, it can be used for a structured applicability assessment regarding the implementation of the health technology in a specific setting (Reference Brereton, Wahlster and Lysdahl7). Finally, the populated logic model should be presented to stakeholders to determine the plausibility and the usefulness of the information provided. Stakeholder feedback informs the final version of the logic model.
Application of the Model in Practice: A Structured and Integrated Presentation of All Findings Regarding the Assessment of Reinforced Home-Based Palliative Care
In our case study, the assessment results were extracted from the evidence summaries of the different assessment aspects in step 3. For instance, the issue of “autonomy and shared decision making” was identified by the assessment methods of four different aspects (legal, ethical, socio-cultural, and patient preferences) as highlighted in Figure 3. An applicability assessment of health technologies can be based on the extended logic model for different scenarios, for example, different countries (Reference Polus, Pfadenhauer and Brereton21). Different organizational and structural variables need to be considered, which can enable or impede the implementation of reinforced home-based palliative care. For instance, limited financial resources are a barrier to the implementation of reinforced home-based palliative care for many countries.
Figure 3. Populated logic model on reinforced models of home-based palliative care (highlighted autonomy examples used in step 4).
Step 5: HTA Conclusion
In step 5, a final appraisal committee consisting of the HTA commissioners and other stakeholders consider the HTA results structured by the logic model to inform their conclusions. The appraisal committee's discussion can be structured by applying decision support tools. These tools can be qualitative, such as consensus reaching processes (Reference DeGroot22), quantitative such as MCDA approaches (Reference Wahlster, Brereton and Burns13), or combinations of both. Flexibility in the application of these tools is crucial, taking distinct political decision settings in different countries and evidence needs into consideration.
Application of the Model in Practice: Toward a Decision Regarding Reinforced Home-Based Palliative Care
Lay and professional stakeholders with different backgrounds (11 members of the National Health Service (NHS) End-of-life Commissioning Group such as physicians, service commissioners, and two former family caregivers) joined a “mock” decision meeting in Sheffield, England. The “mock” meeting simulated decision-making using a simple MCDA method (based on the EVIDEM rating methods). First, stakeholders weighted the decision criteria (effectiveness, cost effectiveness, acceptability etc.) based on a generic description of them. Second, stakeholders scored the HTA results on a scale from +5 to -5 to indicate whether the intervention (reinforced home based palliative care) is “significantly better” or “significantly worse” than nonreinforced home based palliative care.
The intention was to obtain transparent, quantitative judgments of the assessment results. Based on these judgements, the participants identified important issues about the use of HTA results. For example, the committee discussion highlighted concerns about the validity of evidence generated in other countries for the local situation. One participant highlighted that “palliative care often means different things” in the United States, as there are “significant cultural and healthcare organization differences between United Kingdom and United States having a big impact here.” A subsequent deliberative discussion on the findings of the HTA could systematically take these aspects (such as external validity of the HTA results) into account (Reference Brereton, Wahlster and Lysdahl7).
DISCUSSION
Decision makers rely on relevant and meaningful evidence to make fair and legitimate decisions about health technologies in their specific context. Assessments of complex technologies particularly require the consideration of context, implementation as well as patient characteristics and a presentation of the findings in an integrated way rather than side-by-side. The INTEGRATE-HTA Model describes a structured process for such integrated assessments.
The case study adopted a societal perspective and thus included the perspectives of multiple stakeholders. The participation of relevant stakeholders including patients throughout the INTEGRATE-HTA Model enhances the relevance, validity, comprehensiveness, and potential usefulness of the results to the end users (patients, families, healthcare professionals, and decision makers/payers). The comprehensive and transparent presentation of the findings allows for structured discussions and prepares a systematic appraisal, as described in step 5 of the INTEGRATE-HTA model. The weighting exercise in step 5 indicated the priorities of the decision makers. These priorities are important in integrating the different assessment aspects in an HTA that is meaningful for decision makers.
In addition to these strengths, the approach also has some limitations. The INTEGRATE-HTA Model was only applied to one example (the case study on palliative care) which happened in parallel to individual assessment methods being developed. Moreover, the model was further refined throughout the application in the case study. In “real life” applications, the interactive application of the INTEGRATE-HTA Model might prolong the overall HTA timeframe. Its application requires expertise in many fields and coordination between different methodologies. Successful integration of the assessment results relies on agreement about terminology and definitions used within the HTA. For instance, the definition of the socio-cultural context needs to be consistent with, and used within, the assessment methods for context and implementation (Reference Pfadenhauer, Rohwer and Burns9) and for socio-cultural aspects (Reference Lysdahl, Mozygemba and Burns8).
The challenges for integration are multi-dimensional and require flexibility in the application of the INTEGRATE-HTA Model. An initial assessment of the complexity of a technology (through an assessment of complexity characteristics) might be helpful to decide whether all five steps or only some of them should be applied (Reference Lysdahl and Hofmann23). It may be appropriate not to undertake specific assessments for every single aspect, depending on the technology, the scope of the HTA, or the decision-making context. On the other hand, there might be additional aspects that have not yet been considered (e.g., a geographical assessment that would be important for health interventions addressing air pollution). In principle, the INTEGRATE-HTA Model is designed to be applied in assessments of a broad range of curative and preventative health technologies in various healthcare systems and settings. Bond and Weeks showed the potential of the INTEGRATE-HTA Model by assessing in-center and in-home dialysis modalities for the treatment of end-stage kidney disease in Canada (Reference Bond and Weeks24).
CONCLUSION
Issues of integration will become more relevant with the increasing complexity of health technologies. Integration is a process that needs to start from the beginning of planning an HTA. The challenges in integration are multi-dimensional and require flexibility. The INTEGRATE-HTA Model describes a structured process for HTAs considering the realities, in which the technology is being adopted. It contributes to a transparent assessment process and a better understanding of complex health technologies within the system, in which they interact. Stakeholder involvement in all HTA steps is essential to generate a shared understanding and meaningful evidence.
Depending on the technology, the scope of HTA and the decision-making context it may be appropriate not to undertake specific assessments for every single aspect. The comprehensive and integrated output of the INTEGRATE-HTA Model has significant potential to assist health policy makers in achieving fair, accountable, and justified decisions, but requires further testing across a broader range of health technologies and contexts. As the model is applied to other health technologies, a focus should be on assessing its compatibility with the current assessment procedures of different HTA-agencies and on evaluating its acceptability to different stakeholders.
CONFLICTS OF INTEREST
The authors declare that they have no competing interests.
