THE INTEGRATE-HTA PROJECT: ADDRESSING THE CHALLENGES OF COMPLEXITY FOR HEALTH TECHNOLOGY ASSESSMENT
Conceptualization, development, evaluation, and synthesis of complex technologies pose challenges for primary research as well as systematic reviews and health technology assessments (HTA) (1;Reference Brereton, Wahlster and Lysdahl2). The European Union-funded integrated health technology assessment for the evaluation of complex technologies (INTEGRATE-HTA) project (www.integrate-hta.eu) aimed to develop concepts and methods for HTA to enable a patient-centered, integrated assessment of the effectiveness, the economic, social, cultural, legal, and ethical aspects of complex technologies, which takes context and implementation into account.
Within the INTEGRATE-HTA project we conducted a so-called demonstration HTA, a case study in the field of palliative care, to illustrate that application of the concepts and methods developed as part of the INTEGRATE-HTA project are feasible (Reference Brereton, Wahlster and Lysdahl2). Palliative care, underpinned by a holistic, family-centered, individualized philosophy of care, presents an example of a highly complex technology. The case study focused on models of home-based palliative care (HBPC) with and without an additional component of carer support (respectively known as reinforced (rHBPC) (Reference Gomes, Calanzani, Curiale, McCrone and Higginson3) and nonreinforced (HBPC) home-based palliative care). Caregiver support is important, as many patients prefer to be cared for, and die, in their own home (Reference Gomes, Calanzani, Curiale, McCrone and Higginson3). The majority of long-term care at home is provided for by lay caregivers, mostly family members (Reference Carmeli4). This can increase burden for lay caregivers, leading to negative physical, psychological, and social effects (Reference suffering5). It is, therefore, important that HBPC models also provide support for lay caregivers.
Our demonstration-HTA on rHBPC was conceptualized as a generic HTA conducted across seven countries (i.e., England, Germany, Italy, Lithuania, the Netherlands, Norway, and Poland). As implemented, it resulted in a major focus on England, where most evidence and involved stakeholders were located (Reference Brereton, Wahlster and Lysdahl2). Palliative care systems across Europe vary greatly. For example, in England, the palliative care system is well established and HBPC is heterogeneously provided all over the country, mainly through different nongovernmental organizations that often receive support from the state. In Germany, several professional public and private agencies are responsible for ambulatory palliative care services focusing on pain, medical, and psychosocial needs. HBPC care is delivered by volunteers paying home visits to the families and through official ambulatory palliative care services. The reinforced component is, as in England, not an explicit separate component of care but integrated into the service, when professionals visit their patients and thereby also engage with the families functioning as caregivers. In Poland, palliative care is mainly provided in so-called home hospices financed mostly from public sources and normally includes psychological support for the patients’ families.
The differences in how palliative care is provided in England, Germany, and Poland highlight that it cannot be taken for granted that the technology assessed and the findings obtained from a generic HTA are relevant and applicable in the decision-making context of a specific country. At the same time, it is not possible to conduct HTAs on each subject in each country, as is often the case in the field of public health and healthcare due to limited resources and capacities. Therefore, it is critical to examine to what extent the technology assessed and the findings obtained in the demonstration-HTA are applicable in specific European countries.
ASSESSING THE APPLICABILITY OF THE HEALTH TECHNOLOGY
Applicability is defined as the extent to which a technology could be implemented in a specific setting. This includes considering the feasibility of implementing the technology and the variation in intervention fidelity, population characteristics, context, culture, values, and preferences (Reference Cambon, Minary, Ridde and Alla6–Reference Wang, Moss and Hiller8). Applicability assessments can also shed light on the need to adapt or tailor a technology to allow implementation in a specific setting. Indeed, a systematic review of barriers to and facilitators for the use of evidence by policy makers found that over a third of the included studies mentioned use of informal evidence such as local data or tacit knowledge (Reference Oliver, Innvar, Lorenc, Woodman and Thomas9).
Several tools allow, at least to some extent, an evaluation of applicability. These include the applicability and transferability assessment of public health interventions (Reference Wang, Moss and Hiller8), the European network for Health Technology Assessment (EUnetHTA) HTA adaptation toolkit (Reference Chase, Rosten, Turner, Hicks and Milne10), and The Promoting Action on Research Implementation in Health Services (PARIHS) framework (Reference Rycroft-Malone11). However, these tools often involve lengthy and complex processes, such as the conduct of additional context-specific systematic reviews on questions beyond effectiveness or DELPHI processes (Reference Wang, Moss and Hiller8) or do not provide information on how to retrieve local context data (Reference EUnetHTA12). Furthermore, the broad concept of context (Reference Pfadenhauer, Gerhardus and Mozygemba13) is often reduced to the organizational context or very few select parameters and, therefore, does not capture the whole spectrum of what context entails, in particular with respect to complex health technologies (Reference Wang, Moss and Hiller8;Reference Rycroft-Malone11;Reference EUnetHTA12). None of the existing approaches seemed feasible to be implemented within the INTEGRATE-HTA project. We, therefore, set out to develop a pragmatic and easy-to-implement consultation guide to retrieve local context information regarding the applicability of the health technology in question in a given decision-making context.
With regard to the applicability of complex technologies, such as (r)HBPC, context and implementation play a major role in relation to both reach and effectiveness (Reference Wells, Williams, Treweek, Coyle and Taylor14). We made use of the Context and Implementation of Complex Interventions (CICI) framework, a conceptual framework that allows for the comprehensive and structured conceptualization, assessment, and reporting of context and implementation and also developed within the INTEGRATE-HTA project (Reference Pfadenhauer, Gerhardus and Mozygemba13). The CICI framework divides the two dimensions context and implementation into specific domains: eight for context (i.e. geographical, locational, epidemiological, socio-cultural, socio-economic, ethical, legal, political context) and four for implementation (i.e., provider, organization and structure, funding, and policy). The CICI framework has guided the demonstration HTA in terms of taking context and implementation into account and has been applied to different complex health technologies, such as technologies to reduce lead in consumer products and drinking water and technologies to reduce ambient air pollution (Reference Pfadenhauer, Gerhardus and Mozygemba13).
OBJECTIVES
We developed a generic consultation guide to assess the applicability of a health technology in a specific decision-making context in a rapid and pragmatic way. We piloted the consultation guide using the demonstration HTA on (r)HBPC in three European countries to explore whether the guide could be rapidly deployed and deliver useful insights.
METHODS
The study included the following steps, which were all embedded in the demonstration-HTA: (i) development of a consultation guide to assess applicability of the technology, based on the CICI framework and populated with findings from the demonstration-HTA; (ii) piloting of the consultation guide to assess the applicability of rHBPC in England, Germany, and Poland; (iii) analysis of findings and revision of the consultation guide.
DEVELOPMENT OF THE CONSULTATION GUIDE
We developed a consultation guide to structure a conversation with experts who are knowledgeable about the subject matter in their decision-making contexts, posing questions of relevance to the applicability of a health technology. The consultation guide was developed both on a theoretical and empirical basis. The generic structure of the consultation guide and affiliated general and concrete questions are based on the eight context and four implementation domains of the CICI framework (Reference Pfadenhauer, Gerhardus and Mozygemba13), and potentially applies to any health technology. These questions were supported by technology-specific example questions, making use of rHBPC-specific HTA-findings derived from a quantitative review assessing the effectiveness of (r)HBPC (Reference Lysdahl, Mozygemba and Burns15) and a qualitative systematic review of contextual enablers and barriers to the implementation of rHBPC (Reference Brereton, Wahlster and Lysdahl2).
The general structure of the consultation guide is shown in Table 1. It starts with a generic question to ask the experts about factors that can impede or facilitate the implementation of the respective technology. Answers related to the open question inform the twelve domains of the CICI framework. Concrete questions serve to retrieve further details and information of the domains that the expert did not address him/herself. If needed, and possible, the findings of qualitative and quantitative systematic reviews conducted within the original HTA can inform example questions used in the consultation process. Flexible follow-up questions are adapted according to the information provided by the expert, focusing on specific solutions to either overcome barriers or strengthen facilitators located within a given domain. The structure of the consultation guide allows flexible use, for example, the elaboration on points raised during discussion and use of concrete and example questions if necessary.
Table 1. General structure of consultation guide
a Replace wording in italics with description of used technology.
PILOTING OF THE CONSULTATION GUIDE
To apply the consultation guide, we considered the feasibility of and associated advantages and disadvantages of different options for consulting experts in the three countries. In England and Germany, we individually consulted two experts (one highly respected academic expert and one practitioner) with extensive knowledge of palliative care in their respective country. In Poland, an upcoming national palliative care conference facilitated the organization of an expert panel with seven palliative care experts (five clinicians, a nurse, and a social worker). We identified the experts through the INTEGRATE-HTA project's palliative care contacts in each country, each of them well-connected with the national and regional professional organizations (e.g., the German Association of Palliative Medicine, and one of the major church-based organizations, “Malteser”, in Germany; a National Health Service service providing home care for patients with palliative and end of life care needs in England; and the Polish Association of Palliative Medicine in Poland), as well as through the professional organizations themselves contacted by means of email and telephone. All of the identified experts were part of the national palliative care network and had long-standing expertise in the field.
Ethical approval requirements for interacting with stakeholders within the INTEGRATE-HTA project varied by country; local co-ordinators in the seven countries facilitated the Institutional Review Board process and ensured that ethical approval was granted before commencement. The applicability assessment was located within this broader research on stakeholder engagement (Reference Brereton, Wahlster and Lysdahl2). For this additional step, we obtained a waiver from the Ethics Committee of the Ludwig-Maximilians-University in Munich, Germany, under the condition that anonymity of the experts would be ensured.
England and Germany
One researcher (S.P.) conducted the consultations in each country, three by telephone and one face-to-face meeting. The consultations lasted for approximately an hour, ranging from 55 to 90 minutes and were conducted in the native language of the respective expert. We used the consultation guide to retrieve as much information as possible from the first general question. We presented information retrieved from the systematic reviews that had informed the consultation guide as late as possible in the consultation to prevent these results from biasing insights offered by the experts.
The following briefing information was provided before the consultation: (i) A short introduction to the INTEGRATE-HTA project, (ii) A definition of rHBPC as used in the HTA, and (iii) The aim of the consultation including a brief introduction and explanation of CICI framework domains
At the end of the consultations, experts were asked to provide feedback on the consultation guide and its feasibility for assessing the applicability of health technologies in general and (r)HBPC in particular. We audio-recorded the consultations and subsequently transcribed and anonymized sections of interest (Reference Bogner, Littig and Wolfgang16). German consultations were subsequently translated into English.
Poland
The panel discussion lasted for 4 hours and was conducted in connection with a national conference in Poland. Each participant had a printed consultation guide in English in front of her/him. The information was also available on English PowerPoint slides, visible for everyone to follow. The discussion was held in Polish, but the answers were recorded in English on paper (all participants were proficient in English). A previously trained palliative care expert acted as the panel moderator. The experts were encouraged to discuss the issues raised for each domain included in the consultation guide and to share broader ideas. All information was concurrently collected, summarized, and presented on a PowerPoint sheet for approval by the panel. Each participant completed a short feedback form about the process to inform the feasibility and experience of this form of applicability assessment.
Analysis of Findings
We used the CICI framework domains as the overarching structure for our analysis. One researcher (S.P.) used Microsoft Excel to document the information for each domain (i.e., as provided in the transcripts of the individual consultations and the summary sheet of the panel discussion) in an iterative process to ensure that the information retrieved would be assigned to the most appropriate domain. Within each domain, we distinguished between information on the barriers and facilitators and potential solutions. For validation, the summary of each consultation was sent to the respective expert and the panel moderator to ensure that the information was adequately presented and no major issues were missing. The audio-recordings were destroyed once the summaries were finalized.
We examined the findings in terms of depth of insights (i.e., details of information, explanations), breadth of information (i.e., variety of information) as well as resource consumption (e.g., time, financing). Finally, we examined the feedback we retrieved from the individual consultations and the feedback forms collected as part of the panel discussion to examine the experts’ experiences with the consultation guide and the consultation itself. This, as well as the content information retrieved, was used to revise the consultation guide as well as to give feedback to the underlying CICI framework, which was still under revision at the time (Reference Pfadenhauer, Gerhardus and Mozygemba13).
RESULTS
In the following, we summarize the findings of the consultation on the applicability of rHBPC in the three countries and subsequently present the findings regarding the consultation guide as a method to assess applicability.
Applicability Assessment of Reinforced Home-Based Palliative Care
We retrieved extensive information on context and implementation barriers and facilitators that may influence the implementation of rHBPC in the three countries. In Table 2, we present succinct information for each domain of the CICI framework and country. Supplementary Tables 1–3 contain detailed country-specific results.
Table 2. Short Overview of Barriers/Facilitators Mentioned during Applicability Assessment
In all three countries, experts had difficulties with the term “reinforced.” The nature of palliative care means that some provision is usually made for carers, making it difficult to discern whether this constitutes “reinforced” care as the “support” provided in reinforced interventions can be diverse. Likewise, experts in all three countries articulated the problems of limited financial resources and inadequate organization of palliative care services for rHBPC. In Germany and England, (r)HBPC is particularly difficult to deliver in rural areas. While sharing many of the same problems, experts from the three countries placed different emphasis on specific barriers as well as potential solutions. In Germany, for example, the scattered availability of (r)HBPC throughout the country was stressed and advancement in technology (e.g., tele-medicine) as well as a re-organization of home-based palliative care teams were mentioned as possible solutions to guarantee access to (r)HBPC.
Experts from Poland mentioned that the population lacks knowledge and understanding about what palliative care entails. In Poland, panel members suggested that the church, as a key opinion leader, could help improve the negative image of palliative care, for example by providing information during Sunday mass. In England, experts identified the fact that many organizations are involved in the provision of (r)HBPC as a major barrier that impedes smooth implementation. Furthermore, they emphasized that standard procedures, such as referral forms, and good communication help to improve the implementation of (r)HBPC.
Piloting the Consultation Guide
Throughout the consultations, some domains, for which extensive information was identified or which were mentioned repeatedly in the discussion, seemed more important than others. For example, lack of funding was mentioned consistently in all countries as a key factor impeding a holistic implementation of rHBPC. A second major barrier identified in all countries was the organization of the palliative care services. For other domains, experts had difficulties identifying any barriers or facilitators, but this varied between countries. Whereas the experts in England, for example, emphasized the importance of appropriate housing to deliver rHBPC, the expert panel in Poland did not identify any locational barrier or facilitator.
The validation step by the experts in Germany and England and by the expert panel moderator in Poland resulted in no major changes to the summaries. The experts generally confirmed the usefulness of the consultation to identify barriers and facilitators of rHBPC. It was, however, much easier and more natural for participants to focus on and identify barriers instead of facilitators. Overall, the reported feedback of the participants in the pilot study was consistently positive in terms of their understanding of the topic, the specific questions being asked and the purpose of the consultation. The feedback prompted very minor changes to the consultation guide with respect to wording, for example, the description of palliative care, and structure, for example, the order of explanations. These suggested changes were incorporated in a revised consultation guide adapted to rHBPC (available on request).
The involvement of experts with heterogeneous academic and clinical backgrounds in each consultation provided insight into different perspectives and topics. Individual consultations allowed flexible, ongoing interaction between the expert and the interviewer. The insights gathered through the panel discussion did not vary as much as in the individual consultations and produced richer results. This may be the result of the lack of opportunity for peer discussion and agreement in individual meetings. In the expert panel, administrative and financial costs originating from organizing the meeting with more participants, organizing the meeting venue, travel costs, and the time and administrative burden that it took for all participants to be there were much higher than in the individual expert consultations.
The piloting of the consultation guide contributed to the application and subsequent revision of the CICI framework. Indeed, the applicability assessment highlighted the need to apply the CICI framework in a flexible and adaptable way. For example, the concept of rHBPC, as defined in the demonstration HTA, did not fit the German healthcare system well, where reinforcement (e.g., carer support) is implicitly integrated in every palliative care construct available, although not explicitly integrated as a distinct component of HBPC. This was noted in the consultation guide under the domain organization and structure. While the CICI framework structure was very helpful in structuring the consultation, it was more difficult to assign the findings obtained to the different domains. In fact, we noted overlap between several aspects of the context and implementation dimensions. For example, a lack of providers hindering the implementation of rHBPC could either be assigned to the context domain funding, as a lack of funding causes the human resource problem, or could be assigned to the domain provider in the implementation dimension. The CICI framework was revised as a result of this feedback as well as the applications to other technologies, and now structures the implementation dimension in a different way (Reference Pfadenhauer, Gerhardus and Mozygemba13).
DISCUSSION
Key Findings
Our findings demonstrate that, in England, Germany, and Poland, rHBPC is only applicable to a degree and that adaptation of technology components as well as implementation aspects is necessary to ensure that rHBPC can successfully be delivered. Increased funding, training of providers, and investments in organization and structure are considered prerequisites for a successful implementation of rHBPC in all three countries. Experts, however, also mentioned barriers and facilitators to the implementation of rHBPC that were unique to their specific decision-making contexts. It was not surprising that experts had difficulties with the term “reinforced,” as throughout the conduct of the demonstration-HTA, experiences from consultations with stakeholders, such as patients, families, and health professionals showed that the concept of “reinforcement” of HBPC was difficult to understand as involvement and specific support for family caregivers were an integral part of most palliative care approaches (Reference Lysdahl, Mozygemba and Burns15).
In our case study, the consultation guide was found to be a useful tool to collect information regarding the applicability of rHBPC among experts. The answers provided appear useful for decision makers in terms of helping to decide whether the implementation of a health technology is feasible, which aspects of context need to be targeted to facilitate an introduction of the health technology as well as how a potential technology needs to be tailored to contextual needs. The natural mode of the consultation in the form of a discussion, either in the form of individual meetings or in a bigger group, appears to have facilitated information retrieval and to have increased the experts’ understanding of the purposes of the consultation. It was much easier for all participants to identify barriers than facilitators. This can have two important consequences: ignoring the already existing facilitators may lead to the accidental elimination of a key factor during policy change, or to overlook a relatively simple solution to promote the implementation of the technology through a further strengthening of this key factor.
The consensus that arose in the panel meeting compared with the variable perspectives gathered from the individual meetings is concordant with findings in the literature (Reference Heary and Hennessy17;Reference Fern18). The two approaches could be used differently according to different purposes. For example, individual consultations could be used to collect information on as many different barriers and facilitators as possible to take these forward in a separate decision-making team. In contrast, the group meeting could be used to yield consensus and formulate recommendations for implementation of the health technology. In general, all the consultations were easy to implement, although the expert panel meeting was costlier and more time-consuming than the individual consultations.
STRENGTHS AND LIMITATIONS
We piloted the consultation guide in three distinct decision-making contexts using two different methodological approaches but did not formally test the guide. In particular, we did not validate the tool and only assessed one health technology in the field of palliative care. While we consider it a strength that we conducted the consultations in the experts’ respective native language, we cannot exclude the possibility of loss of information due to subsequent translations. Translations were done by the same researcher conducting the consultations.
We only consulted two experts in both England and Germany and six experts in Poland. Thus, the findings may not reflect the full range of perspectives among experts and organizations. It would also be worthwhile to include a broader range of stakeholders, such as patients, families, and different healthcare professionals in the applicability assessment, depending on the health technology to be assessed and time and resources available. An online DELPHI process might be practical: In several DELPHI rounds, stakeholders could consent on a set of barriers and facilitators relevant for their specific contexts. However, a DELPHI process usually requires significant time and resources to review the literature, program the survey, manage and moderate several DELPHI rounds, and analyze results adequately (Reference Yousuf19). In our case study, given the complexity of the topic (rHBPC) and our concern with many aspects of context and implementation (as structured by the CICI framework), we experienced the more flexible back-and-forth between consulter and expert(s) as an advantage.
While in our case study the identification of appropriate experts was not a problem, it may be time-consuming and challenging in other contexts, especially to ensure that different views, distinct levels of the healthcare system and geographical areas are adequately represented. This may become even more challenging when a greater range of stakeholders is involved.
In this pilot study, we assessed the applicability of an existing rather than a new health technology; experts were, therefore, partly dealing with problems that were already occurring in the countries. Applying the consultation guide to a new health technology in a specific context may be more difficult, involving less actual experience and more speculation.
Compared with other applicability and transferability tools (Reference Wang, Moss and Hiller8;Reference Rycroft-Malone11;Reference EUnetHTA12) our consultation guide may offer a pragmatic and rapid applicability assessment eliciting current experience and tacit knowledge regarding the influence of context during expert consultations. Importantly, this applicability assessment tool can be applied in a very generic manner without discrete empirical information on the technology in question as it takes the generic key features of complex technologies for the implementation in a specific context into account (Reference Lysdahl and Hofmann20). Additionally, it can make use of specific information collected as part of a HTA including economic, ethical, socio-cultural, and legal aspects. In doing so, the applicability assessment can be embedded within the overall HTA process as proposed by the INTEGRATE-HTA Model (Reference Wahlster, Brereton, Burns and Hofmann21). We believe that this approach has the potential to save the costs and time required for conducting a full HTA, and that it can be used flexibly according to the resource capacities available. However, further studies are needed to validate this guide and apply it to other types of health technologies and more diverse decision-making contexts.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: https://doi.org/10.1017/S0266462317000745
Supplementary Table 2: https://doi.org/10.1017/S0266462317000745
Supplementary Table 3: https://doi.org/10.1017/S0266462317000745
CONFLICTS OF INTEREST
The authors have nothing to disclose.
