Stakeholder involvement in health technology assessment (HTA) is advocated internationally (Reference Cicchetti, Iacopino, Carletto, Marchetti and Mennini1). Stakeholders are “individuals, groups, or organizations which not only share the benefits of the topic under scrutiny, but who can potentially affect the goals or the performance of a sector, plan, or policy” (2, p. 85). Hence, stakeholders include lay people (e.g., patients, family members, carers or representatives of patient organizations), professionals (e.g., health and social care staff, academics) and others (e.g., volunteers, support groups) with an interest in the topic. Funding bodies such as the National Institute for Health Research (NIHR) in England and the Research Council in Norway require patient and public involvement in grant applications and throughout the HTA.
Creating opportunities for stakeholder involvement in HTA and providing a “voice” for interested parties in decision making is important (Reference Yazdizadeh, Shahmoradi and Majdzadeh3) as this potentially ensures shared key priorities are addressed and research findings are translated into practice (4). To avoid paying lip service to stakeholder involvement, this needs to be undertaken for a legitimate purpose and be appropriately inclusive for the HTA being undertaken (e.g., including stakeholders from various services) (Reference Nielsen, Lauritsen and Kristensen2). However, including stakeholders in HTA and decision making is challenging as policy and decision makers are faced with swift technological developments and the increasing requirement to provide applicability assessments of complex health technologies (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5). The situation is confounded by increasing pressures to demonstrate transparency in decision-making processes (Reference Nielsen, Lauritsen and Kristensen2). Some reluctance to involve stakeholders has been reported (Reference Nielsen, Lauritsen and Kristensen2), perhaps due to concerns about introducing bias to the HTA process (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5).
Despite these challenges, examples of stakeholder involvement exist in some HTAs, for example, cochlear implant, albeit that differences in stakeholder views give rise to continuing debates about ethically contested issues (Reference Daniels and van der Wilt6). Indeed, little guidance exists about stakeholder involvement in HTA, especially with regard to those with rare diseases or affected by sensitive issues, such as palliative care. That said, some guidance has been published about patient and public involvement in health and social care research by INVOLVE, in the United Kingdom. (7). INVOLVE is funded by the National Institute for Health Research (NIHR) in England. Additionally, Popay and Collins (Reference Popay and Collins8) have published guidance for evaluating the impact of public involvement in research. Although useful, such guidance does not make reference to other stakeholders. Hence, given policy and funder expectations, several questions arise about stakeholder involvement, notably who to involve, when and how to involve them (Reference Abelson, Giacomini, Lehoux and Gauvin9). Additionally, questions also arise about the value and impact of stakeholder involvement throughout HTA (Reference Pizzo, Doyle, Matthews and Barlow10).
We involved palliative care stakeholders in a large European project (INTEGRATE-HTA) that developed concepts and methods for the integrated assessment of complex technologies because policy makers need better tools to support their decision making in this area (see http://www.integrate-hta.eu/). To demonstrate their feasibility and value, the concepts and methods developed in the project were applied in a case study that evaluated models of palliative care service delivery as an example of a complex technology (Reference Brereton, Wahlster and Lysdahl11). We assessed home-based models of palliative care (HBPC) with and without an additional, explicit and intentional component of informal carer “support” (known as “reinforced” and “nonreinforced” models of home care, respectively, after the Cochrane review by Gomes et al.) (Reference Gomes, Calanzani, Curiale, McCrone and Higginson12). Carer “support” may include education, counselling or other supportive interventions.
The palliative care case study demonstrating the application of some of the INTEGRATE-HTA methodological guidances was undertaken in England for pragmatic reasons as palliative care provision differs throughout Europe. The case study was designed to inform the following research question: “Are reinforced models of home based palliative care (rHBPC) acceptable, feasible, appropriate, meaningful, effective, cost-effective models for providing patient-centered palliative care [compared with nonreinforced models of home based palliative care (HBPC]) in adults (defined as those aged 18 years old and over) and their families?” Stakeholder involvement was an important source of evidence in the case study as, following the development of methodological guidance. This study reports on the extensive stakeholder involvement that occurred throughout the palliative care case study and reflects on the successes, challenges and lessons learned from stakeholder involvement at each stage of the HTA.
METHODS
The five-step INTEGRATE-HTA Model (Reference Wahlster, Brereton and Burns13), which enables integration of relevant assessment aspects, was used to guide the application of new concepts and methods in the palliative care case study. Steps 1 and 2 define the scope of the HTA, Step 3 coordinates the assessment of evidence. Steps 4 and 5 structure the applicability appraisal and final HTA recommendation. Stakeholder involvement was identified as important from the outset of the study and €URO 15,000 was available for stakeholder involvement in each country for example, to provide incentives for participation, to cover any costs incurred by individuals. The amount spent in each country varied as some stakeholders declined payment.
Step 1 and 2: Stakeholder Involvement in Scoping the Palliative Care Case Study
Stakeholders participated in scoping for the palliative care case study in seven countries (England, Italy, Germany, The Netherlands, Norway, Lithuania, and Poland). Because policy, philosophy, expectations, and consequently methodological, ethical, and practical issues for stakeholder involvement and palliative care research varies in each country, researchers used locally advocated methods for stakeholder involvement. Hence, either a consultative approach based on the National Institute for Health and Care Excellence (NICE) guidance (14) or a qualitative research approach was used to involve stakeholders in scoping in each country. The methods and findings of the scoping phase have been previously reported (Reference Brereton, Ingleton and Gardiner15).
To identify their perspectives about key issues and topics of importance for palliative care at the outset of the case study, a local co-ordinator (member of the INTEGRATE-HTA team or known associate) led stakeholder involvement in each country. The local co-ordinator ensured that the identification and recruitment of lay people and professionals was appropriate for the local context and approach to stakeholder involvement adopted. Some stakeholders were identified by INTEGRATE-HTA researchers with palliative care experience and knowledge of individuals or groups with lay or professional expertise in the field. The co-ordinator in each country approached known stakeholders directly and sought the assistance of managers or a key professional in services delivering palliative care to identify and recruit previously unknown stakeholders to the project. To ensure that lay people had relevant experience, we recruited them from local palliative care services or groups known to have an interest in, or experience of, palliative care.
We involved a total of 132 stakeholders (82 professionals and 50 “lay” people) aged ≥18 years in all seven countries between May and September 2013. Lay stakeholders include patients’; some with experience as a patient and carer of someone with palliative care needs; carers, ex-carers, family members, and friends of someone with palliative care needs as well as members of palliative care group / associations and volunteers (among lay stakeholders in Poland). Professional stakeholders include clinicians; researchers; staff with a dual clinician and researcher role; managers, social worker, and pastoral care specialist and volunteers (among professional stakeholders in Italy).
Where known, most stakeholders were white females, with lay stakeholders aged 27–89 years and professionals aged 28–69 years. Stakeholders participated in either individual face-to-face or telephone meetings or interviews, group meetings or focus groups (see Table 1). Stakeholder consultation occurred in meetings where information was collected and summarized using the EUnetHTA Core Model® (Reference Lampe and Mäkelä16) as an overarching framework. Individual interviews or focus group meetings were conducted using semi-structured interviews within a qualitative approach in keeping with local tradition and researcher preference. Thematic analysis was used to identify key issues across countries (See Brereton et al. for further details) (Reference Brereton, Ingleton and Gardiner15).
Table 1. Background Details of the Stakeholders Involved in Scoping in Each Country
a In Lithuania, the professionals who were approached to provide information included 3 representatives of patient organisations in lieu of approaching lay stakeholders for two reasons: (Reference Cicchetti, Iacopino, Carletto, Marchetti and Mennini1) patients receiving palliative care have a serious health condition and (Reference Nielsen, Lauritsen and Kristensen2) palliative care in Lithuania is a relatively new field and locals thought competent opinion could perhaps be expressed more by a specialist. Lay stakeholders: P, patient; P&C, patient & carer; C, carer; Ex-C, Ex-carer; FM, family member; Fr, friend; PC Group, member of palliative care group / associations; V, volunteers (among lay stakeholders in Poland). Professional stakeholders: Cl, clinicians; R, researchers; Cl & R, clinician & researcher (dual role); M, manager, SW & PCS, social worker and Pastoral Care Specialist; V, volunteers (among professional stakeholders in Italy).
E, England; G, Germany; I, Italy; Ne, Netherlands; No, Norway; Pn, Poland; L, Lithuania.
Step 3: Stakeholder Involvement in Evidence Collection and Assessment
Between April and June 2015, stakeholders both provided information that informed evidence collection and participated in the review of assessment results as outlined below (see Table 2). Some stakeholders had previously taken part in scoping of the palliative care case study (steps 1 and 2). Lay stakeholders who were members of a local palliative care advisory group or cancer research group in England were invited to participate by the local co-ordinator. Professional stakeholders were either directly identified and recruited to step 3 by members of the INTEGRATE-HTA project team or they volunteered to participate having been given information about opportunities to participate in the research by service managers who distributed information to them on behalf of the INTEGRATE-HTA team. However, time and resource constraints meant that we were unable to involve lay stakeholders (i.e., patients, lay caregivers, or other interested parties) in all assessments.
Table 2: Background details of the stakeholders involved in evidence collection and assessment, the applicability assessment and a ‘mock’ decision making meeting.
Key: E-England; G-Germany; Ne-Netherlands; Pn-Poland; Prof – Professionals (included Clinicians, Researchers and those with a dual Clinician & researcher role); Lay – Lay person(s) (included P:Patient; P&C: patient & carer; Ex-C: Ex-Carer; FM: Family member).
Economics
A consultative approach was used to involve stakeholders in the economic assessment (Reference Lysdahl, Brereton and Oortwijn17) and several methods were used to elicit information including email communication, telephone discussions, face to face meetings, and workshops. Nine stakeholders (eight professionals and one female lay person with experience as a patient and carer) contributed to the two workshops for the economic evaluation in England, both guided by a semi structured consultation guide. In the first workshop stakeholders provided information that informed an understanding of the problem and conceptual modelling. In the second workshop, stakeholders provided data to fill gaps in the economic model and discussed the results of the economic analysis.
Socio-cultural
Some steps from the INTEGRATE-HTA guidance to assess socio-cultural aspects (Reference Lysdahl, Brereton and Oortwijn17) of HBPC and rHBPC, specifically user-professional relationships, were applied through consulting nine stakeholders in England using semi structured consultation guides. Two researchers (one with sociological and health sciences expertise (German) and one with palliative care expertise (English native speaker) were involved. Two professional stakeholders participated in individual consultations lasting approximately an hour by means of telephone. Subsequently, one face-to-face group consultation, lasting approximately 2 hours, took place with four lay stakeholders (one female with experience as a patient and carer; one female relative and two male ex-carers) in England initially using an open question guide. At the end of the meeting, a prioritization exercise took place. The same researchers completed a second group consultation, lasting approximately an hour, with three professionals in England by means of video-Skype to gain a deeper understanding of “the user-professional-relationship and decision making.” A participatory approach was used primarily to test if stakeholders could apply the typology developed in the socio-cultural framework within the consultation. Before the meeting, documents about the content of the consultation (including the framework to be discussed) were shared.
Ethical
The information provided by stakeholders in the scoping phase of the case study, socio-cultural and economic assessments was subsequently used to inform an assessment of the complexity of, and ethical issues associated with, (reinforced) HBPC by one researcher using a procedural framework devised by Lysdahl et al. (Reference Lysdahl, Brereton and Oortwijn17).
Effectiveness
To compare reinforced with nonreinforced HBPC interventions, the systematic review by Gomes et al. (Reference Gomes, Calanzani, Curiale, McCrone and Higginson12) was updated as part of the INTEGRATE-HTA guidance to assess effectiveness aspects (Reference Burns, Chilcott, van Hoorn, Kievit and Rehfeuss18). Harvest plots were created to portray heterogeneous evidence in a clear, transparent way as meta-analysis was less appropriate. The harvest plots were used as a basis for a gap-analysis of the existing literature, which was subsequently used to inform individual semi-structured telephone consultation with four professional stakeholders (including male and female researchers and practitioners) from three countries (England, Germany, and the Netherlands). No lay stakeholders were involved in the effectiveness assessment due to time and resource constraints.
Patient Preferences and Moderators of Treatment Outcome
A summary of the results from the assessment of both patient preferences and moderators of treatment outcome was presented to an advisory panel of seven stakeholders in England that consisted of two male ex-carers, aged 63 years and 74 years and five experienced palliative care nurses working in a range of settings. The meeting took place by means of video-Skype with researchers based in the Netherlands and the support of a researcher based in England who was present in the group meeting. Using a semi-structured consultation guide, stakeholders were asked to validate the assessment findings and to provide an explanation for these or additional insights.
Legal
There was no stakeholder consultation in the legal assessment.
Step 4: Stakeholder Involvement in Appraisal/Applicability
Applicability Assessment
An applicability assessment was undertaken with twelve professionals (two in the United Kingdom, two in Germany, and seven in Poland) to identify contextual and implementation barriers and enablers specific to the application of the demonstration HTA findings. All professional stakeholders were directly identified and recruited to step four by members of the INTEGRATE-HTA project team or their known associates who assisted with the project locally. No lay stakeholders were involved in the applicability assessment due to time and resource constraints. In the United Kingdom and Germany, one researcher with HTA experience conducted individual meetings using a consultation guide by telephone, except for one professional, who participated in a face-to-face meeting. The consultations lasted approximately an hour, ranging from 55 to 90 min. In Poland, a previously informed palliative care expert facilitated a panel consultation lasting approximately 4 hours with eight professionals. The professionals were encouraged to discuss the issues raised for each domain identified in the consultation guide and provide additional information. All information was concurrently collected, summarized, and presented on a PowerPoint Sheet for validation by the expert panel.
Step 5: Stakeholder Involvement in Decision Making
Step 5 results in the HTA conclusion and recommendations. A group of thirteen stakeholders (11 professionals involved in commissioning end of life services and two lay stakeholders in England) participated in a mock decision-making meeting. Professional stakeholders were recruited to step 5 by the chair of the commissioning group who was approached directly by the local co-ordinator in England. Lay stakeholders (one male and one female, both ex-carers) were members of the palliative care advisory group who were also invited to participate in the meeting by the local co-ordinator with the agreement of the chair of the commissioning group as the mock decision-making meeting took place instead of the group's scheduled meeting. Multiple criteria decision analysis (MCDA) may be used to support decision making in HTA. MCDA guides stakeholders to weight the assessment criteria (effectiveness, cost-effectiveness, etc.) based on a generic description of them. In the consultation meeting, stakeholder were presented with the results of the assessments. Stakeholders then scored the HTA results on a scale from +5 to -5 to indicate whether the intervention (i.e., rHBPC) is “significantly better” or “significantly worse” than nonreinforced HBPC. Scoring stimulated open discussion amongst group members.
RESULTS
Stakeholder Involvement in Scoping the Palliative Care Case Study
For a detailed report about stakeholder involvement in scoping of the palliative care case study for the INTEGRATE-HTA project, cf. Brereton et al. (Reference Brereton, Ingleton and Gardiner15). We successfully involved a large number (n = 182) of palliative care stakeholders in seven European countries in scoping. In terms of added value, stakeholders identified twenty-three issues that were common to three or more countries. Stakeholder involvement (alongside a review of review level evidence) informed the project's main research question, enabling us to identify “reinforced” (rHBPC) and “nonreinforced” (HBPC) models of home care as the intervention and comparator of interest. Although reinforced models of home palliative care explicitly address only two of the issues raised by stakeholders in several countries (i.e., the need to increase home care and the need for caregiver training/support), researchers can be confident that the intervention is important to both lay and professional stakeholders internationally.
The remaining findings increased the research team's awareness of key issues in palliative care as some HTA researchers had limited experience in the field. The information provided also assisted the development of sub questions for use in the case study (e.g., for the socio-cultural aspects which focused on the user-professional-relationship and decision making). Additionally, we subsequently used stakeholder information provided during the scoping phase to inform a logic model (Reference Rohwer, Pfadenhauer and Burns19) and specific assessments, notably the complexity and ethical assessments related to HBPC within the HTA.
Key challenges exist in ensuring inclusivity of all stakeholder groups, for example, enabling stakeholders who are very ill to contribute to HTA. Additional challenges exist in terms of identifying stakeholders because some provide insights from the perspective of both a patient and carer, having fulfilled both roles and others, notably volunteers, were identified as professionals in some countries and lay people in others. The use of different approaches to stakeholder involvement across countries posed several challenges, notably for the collective analysis of information / data. Not only does the terminology for consultation and qualitative research differ but there is a need for guidance about how to analyze the collective findings from these different paradigms.
We learned that the underlying philosophy and views of appropriate and feasible methods of stakeholder involvement vary internationally, resulting in different ethical requirements and practical considerations. However, stakeholders, including patients and families undergoing palliative care, can inform project decision making. Methods for stakeholder involvement, especially consultation, need further development.
Stakeholder Involvement in Evidence Collection and Assessment
We successfully involved several lay and professional stakeholders in evidence collection and assessment of HTA findings for several aspects of the palliative care case study. Irrespective of the range of face–face or remote (e.g., telephone, video-Skype) methods and activities used, when both lay and professional stakeholders participated in meetings, they worked well together. Stakeholders added value to the HTA by drawing on their experiential knowledge to provide additional insights for researchers in terms of informing the assessment; prioritizing issues; informing gap analysis, validating and interpreting assessment findings. For example, stakeholders provided economic information that is not readily available elsewhere, making their involvement in a series of workshops and meetings worthwhile.
Stakeholder consultations provided fresh perspectives on the effectiveness evidence, which indicated no effect of reinforced care compared with nonreinforced care for most patient and carer outcomes. They also highlighted implications for practice and further research. For example, stakeholders emphasized the need to develop a clear understanding of nonreinforced (i.e., usual care) as the nature of palliative care means that this is likely to be tailored to provide individualized, holistic care for the family. Such understandings are important to identifying what alternative, additional services may be effective. They also suggested alternative evaluative designs are needed in palliative care research as the outcomes frequently measured may not reflect the purpose of interventions such as reinforced home care.
Although stakeholder involvement in some assessments (e.g., effectiveness) did not create much additional work for researchers or stakeholders, it was a resource intensive exercise in terms of time for both stakeholders and researchers in some other assessments (e.g., economics). Some assessments (e.g., the socio-cultural assessment) proved challenging because they involved several steps and different stakeholders in each step. In such situations, stakeholders require careful preparation with regard to information they receive in advance of the meetings.
Several lessons were learned, notably that interactive forms of communication were the most productive forms of stakeholder involvement. Successful stakeholder involvement probably relies on the perceived relevance of the decision problem and requires those taking part to be enthusiastic and committed to become, and stay involved, especially when more than one meeting or workshop is planned. Careful planning is required to ensure stakeholders receive sufficient information and are adequately prepared for each involvement activity.
Stakeholder Involvement in an Applicability Assessment
Appraisal / applicability assessments were successfully completed using various methods (face–face and telephone meetings or panel discussions) in three countries. The findings added value by indicating that organizational and structural barriers need to be considered in all three countries to ensure the applicability of rHBPC. The underlying issue for many of the stated barriers concerning the implementation of rHBPC is the limited availability of financial resources. Involving stakeholders in a group meeting provides the opportunity to validate findings with experts; assists in assuring the quality of information used in the applicability assessment and may provide deeper insights into the applicability of the assessed intervention. However, the depth of retrieved information does not necessarily decrease by consulting experts in one-by-one consultations.
A key challenge is identifying relevant stakeholders who have sufficient time to attend a lengthy meeting for the applicability assessment (when panel meetings are used). Again, we learned that various methods can be used to involve stakeholders in applicability assessments.
Stakeholder Involvement in Mock Decision Making
We successfully engaged both lay and professional stakeholders in a final “mock” decision-making meeting. Although a decision could not be finalized, stakeholders still added value by highlighting several important issues related to the benefit of rHBPC and the relevant evidence in the case study.
The final mock decision-making meeting posed many challenges as a large amount of information needed to be summarized and discussed in a relatively short time. Similarities between the intervention and comparator make them difficult for some stakeholders to differentiate even though working definitions of the terms “home based palliative care” and “reinforced home based palliative care” were provided. Although the differences in the intervention and comparator were articulated by an experienced practitioners in the meeting, some stakeholders believe that current practice includes some provision for carers and, therefore, “reinforced” HBPC is being provided (which other stakeholders do not believe is the case). Overlaps between categories in the assessment criteria existed along with interactions between different assessment aspects. External validity, evidence gaps, and the need for a well-defined, clearly differentiated comparator were all issues of concern for stakeholders in the meeting. For final decision-making, these issues should be taken into account.
DISCUSSION
This study reports on the successes, challenges, and lessons learned about stakeholder involvement throughout HTA. Despite successes, and strengths of the project, in terms of our extensive involvement of lay and professional stakeholders throughout the palliative care case study, inclusivity proved challenging as lay stakeholders were not involved in some assessments. The lack of involvement of lay stakeholders in some assessments is undoubtedly a limitation given the increasing calls to access public views throughout HTA (Reference Sampietro-Colom and Thomas20). Furthermore, some groups (e.g., volunteers) were under-represented throughout the case study due to resource limitations and ease of access which reflects findings of a review of stakeholder involvement in program evaluation (Reference Brandon and Fukunaga21). Indeed, similar challenges regarding inclusivity in stakeholder involvement are recognized within the wider literature, which not only gives rise to concerns about ensuring representation of stakeholder interests (Reference Nielsen, Lauritsen and Kristensen2), power (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5), fairness (Reference Nielsen, Lauritsen and Kristensen2;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5;Reference Abelson, Giacomini, Lehoux and Gauvin9), legitimacy (Reference Nielsen, Lauritsen and Kristensen2;Reference Gauvin, Abelson, Giacomini, Eyles and Lavis5), and transparency of decision making (Reference Abelson, Giacomini, Lehoux and Gauvin9).
Despite inclusivity posing challenges, palliative care provides a good exemplar for other health conditions as patient stakeholders’ health will vary throughout the disease trajectory and its holistic nature captures the views of others (e.g., family carers) who are stakeholders in their own right. Furthermore, palliative care allows inclusion of generalist (i.e., practitioners working in palliative care who do not have specialist palliative care qualifications) and specialist health and social care professional stakeholders (i.e., practitioners working in palliative care who have specialist palliative care qualifications). To be inclusive, researchers require a good knowledge of services and the potential stakeholder groups affected by the HTA. Whilst inclusivity of stakeholders from a variety of locations potentially increases the value of the HTA findings, further guidance is needed about how to maximize stakeholder involvement in HTA and how to best acknowledge and address the ethical issues that arise.
Securing such high levels of stakeholder involvement in seven countries and identifying similarities in findings in the scoping phase of the case study provides some confidence that the key issues identified by stakeholders are of shared international importance. Several issues identified by stakeholders, such as the emphasis on home care are reflected in the international palliative care policy and literature (Reference Gomes and Higginson22). The insights gained through the inclusion of heterogeneous stakeholders (e.g., patients, family members, clinicians, and academics) evidences ways that their expertise benefits HTA and researchers involved in HTA. Our case study findings suggest that not only can stakeholders help to identify key issues in the field; inform the choice of intervention and comparator for assessment, provide evidence and assist the interpretation of findings, they can also highlight issues influencing applicability and potentially contribute to decision making. Indeed, heterogeneous stakeholder involvement may be of particular value in the assessment of complex interventions as this can contribute to understanding how the intervention may interact with different health contexts and settings (Reference Sampietro-Colom and Thomas20).
Although some authors refer to the benefits of stakeholder expertise for HTA (Reference Cavazza and Jommi23), most texts refer to “expert” (i.e., professional) and lay stakeholders, implicitly suggesting that professionals as the “experts” have greater, or more valuable, knowledge and insights than lay stakeholders. It seems important to acknowledge that recognize the different expertise of all stakeholders and acknowledge that their experiences, views, and contributions to HTA are equally valuable. That said, further work is needed that clearly illustrates the added value of contributions from each stakeholder group.
Using different approaches (i.e., qualitative research and stakeholder consultation) in the scoping phase of the case study, respected local understandings about the best methods of stakeholder involvement. However, despite similarity in findings across countries in the scoping phase, analysis proved challenging. It is tempting to suggest that there is a need to identify or develop methods of stakeholder involvement that are acceptable internationally, so that the findings from cross country stakeholder involvement can be more easily compared. However, this would fail to take account of differences in healthcare systems and administrative traditions within which HTA organizations function in each country (Reference Cavazza and Jommi23).
Using a range of stakeholder involvement methods throughout the HTA successfully enables flexible and responsive information exchange to ensure common understandings develop. Indeed, no one method of stakeholder involvement will be suitable in all situations, especially when including diverse stakeholder groups, vulnerable or very ill stakeholders. Although, the interactive approaches adopted in this project were selected as pragmatic ways of eliciting information to demonstrate concepts and methods developed in the INTEGRATE-HTA project, they proved useful in demonstrating the feasibility of stakeholder involvement throughout HTA and reflect the principles of good practice for stakeholder involvement in the conduct of HTA (Reference Stephens, Handke and Doshi24). Indeed, involving stakeholders throughout HTA provides the real world data that will complement RCT data (Reference Sampietro-Colom and Thomas20). Feedback from stakeholders involved in our study indicated that everyone felt able to contribute to the discussion and learn from their involvement in the project. Nevertheless, further guidance about stakeholder consultation methods is needed to enhance the quality and effectiveness of stakeholder involvement activities.
CONCLUSIONS
This case study suggests that stakeholder involvement is both feasible and worthwhile throughout HTA. Stakeholders’ experience and expertise can help to identify key issues in the field; inform the focus of the assessment (e.g., interventions, comparators, questions, and sub questions), provide evidence and assist the interpretation of the evidence. Furthermore, stakeholders can highlight applicability issues and potentially contribute to decision making. The immediate benefit for HTA researchers and stakeholders themselves is potentially more widely realized after decision makers have agreed the action, if any, required. Challenges to stakeholder involvement relate to ensuring inclusivity, especially for patients and the public, and overcoming resource limitations, notably with regard to the time required for such activities in some assessments.
We learned that views of the best methods of stakeholder involvement and the associated ethical requirements vary internationally and that various methods can be used to involve stakeholders throughout HTA. With careful planning, lay and professional stakeholders can be involved throughout HTA and the impact of their involvement on the project, researchers and stakeholders themselves could be assessed. To minimize potential challenges, researchers need to develop some understanding of the local context for stakeholder involvement to be able to comply with ethical requirements, adopt appropriate methodologies and address practical issues related to stakeholder involvement. Further guidance about stakeholder involvement throughout HTA is required, especially for vulnerable or hard to reach groups. We recommend that the experiential and colloquial knowledge provided by stakeholders should be recognized by researchers as part of the evidence hierarchy and methods developed to evaluate the impact of stakeholder involvement in HTA, especially in areas where both the quantity and quality of the evidence is limited.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
