Introduction
Patients are in a unique position to contribute an essential perspective to health technology assessment (HTA) as they know what it means to live with a condition and to be treated for it. Besides being “carriers of data,” allowing them to inform researchers about the symptoms and adverse events that matter most to them and have the greatest impact on their lives, patients can also contribute to the actual assessment process as co-researcher. This could contribute to the relevance and feasibility of the research questions tackled in the HTA.
Several HTA agencies have built up experience with patient involvement (PI) in HTA, for example, HIQA (1), CADTH (Reference Berglas, Jutai, MacKean and Weeks2), NICE (3), and G-BA (4). Also EUnetHTA, the European Network of HTA agencies, has involved patients in its joint or collaborative assessments (5), and the Spanish Network of Health Technology Assessment, RedETS, has developed a guideline for PI in HTA (Reference Toledo-Chávarri, Perestelo-Pérez, Álvarez-Pérez, Abt-Sacks, Santoro Domingo and Villalón6). In 2019, PI in HTA at the Belgian Health Care Knowledge Centre (KCE) was limited to the representation of the three Belgian patient umbrella organizations (Dutch, French, and German-speaking) in the Board and unsystematic involvement in HTAs as part of the stakeholder consultations.
Awareness of the particular role patients could play in HTA and other research performed at KCE led to desire to develop PI procedures at KCE. It was considered necessary, however, to develop a formal organizational position with respect to PI in HTA first, before developing and implementing organization-wide PI procedures. An organizational change can only be successful if it takes the culture of the organization toward this change into account (Reference Renedo, Marston, Spyridonidis and Barlow7). Moreover, measuring the organizational culture can as such prepare the organization for the future implementation of PI.
This study describes the culture within KCE toward PI in HTA, with the aim to use this information in the development process of organizational position statements about PI in health policy research at KCE.
Methods
Context
The initiative to develop organizational position statements about PI in health policy research was taken by the management of KCE. A topic proposal to develop PI processes at KCE had been accepted by the Board of KCE, and soon after the start of the process development project, it was felt necessary to be explicit about KCE's position with respect to PI in health policy research, because the processes to implement depend on the level and intensity of PI that the organization endorses. The development of the position statements was considered a full-fledged KCE project, although its contents were not scientifically validated by external validators like other reports, because positions can hardly be contested based on scientific arguments. The report was submitted for approval to the Board, which consists of different stakeholders in the Belgian healthcare system.
The project team consisted of three qualitative research experts, one health economist, one expert with a nursing background, and the general manager of KCE, who has a background in health economics and philosophy.
Scope
KCE performs, besides HTA, also other types of health policy research, such as health services research (HSR), good clinical practice guideline development, pragmatic clinical trials, and methodological research. The results for the other types of policy research are often also applicable to HTA. We report on all results relevant and applicable to HTA.
The focus is on the involvement of patients in HTA, that is, people affected by the disease under consideration, or their representatives if patients cannot express themselves, rather than lay people from the general public, although it is recognized that for some topics, for example, prevention, “patients” can include people not (yet) affected by a disease.
Concept definitions
PI in HTA is defined as doing assessments “with” and “by” patients (8). It could encompass, for example, involvement in the choice of HTA topics, helping to define the scope of the HTA, assisting in the design, assessment, or dissemination of the findings of the HTA.
PI in HTA encompasses several intensities of engagement. We complemented the conceptual definition of PI of INVOLVE with the operational definitions developed by Hughes and Duffy for public involvement in research and adapted the latter to the more narrow focus of PI in HTA: (Reference Hughes and Duffy9)
• Targeted consultation implies involvement where patients are contacted and consulted on specific aspects of the HTA, for example, to provide feedback on a summary or on the wording of a questionnaire (Reference Hughes and Duffy9).
• Embedded consultation is a level of involvement where patients are consulted regularly throughout the entire HTA process, from developing ideas for assessment of topics to disseminating findings. The HTA team keeps ownership and control over the assessment (Reference Hughes and Duffy9).
• Collaboration and co-production implies involving patients in the assessment team, either as co-researchers or as contributors to key decisions regarding the HTA processes and findings, for example, about the tools, choice and wording of survey questions, data analysis, and presentation of findings (Reference Hughes and Duffy9).
• User-led involvement or coordination implies that patients take the lead, control, and manage the HTA. They decide on the scope, research questions, design, planning, and reporting of the HTA (Reference Hughes and Duffy9).
Culture measurement
The flow-chart of the study, with the different steps taken to arrive at position statements about PI at KCE, is presented in Figure 1. First, we measured the culture toward PI at KCE, then we developed draft position statements which were subsequently submitted to the Delphi panel in two rounds. An open online survey was then sent to all KCE staff members to obtain written comments on the draft position statements, which helped to formulate the final statements. Participants were from the beginning fully informed about the reasons and ultimate objective of the exercise, that is, to develop position statements about PI at KCE.
Figure 1. Development process of the position statements about PI at KCE.
The openness/resistance toward PI in KCE research, including HTA, was measured using a nominal group technique. A board game was developed as data collection instrument for the nominal groups (Figure 2). The “patients on board”-game was played by researchers at KCE as well as by the supporting and administrative staff and management, in eight groups of six to eight people: one for HTA (n = 6), two for HSR (n = 12), one for clinical practice guidelines (n = 6), one for pragmatic trials (n = 6), two for support (n = 16), one for data analysis (n = 6). The rationale for creating different groups for different types of research or roles in KCE projects was that it would facilitate the communication amongst the participants because they understand very quickly what someone is talking about. Groups started at different phases of the research process, because time was limited and the aim was to collect arguments for all research phases. Each group played simultaneously in separated rooms under the supervision of a game master. The game masters were all internal to KCE and included all members of the research team, one communication specialist, one project facilitator, and one program manager.
Figure 2. KCE's “Patients on Board”-game as a nominal group instrument.
The aim of the game was to collect arguments for or against PI or conditions for PI in different phases of the research process: identification and selection of topics for research, scoping, design and methods definition, data collection and analysis, reporting of results, formulation of recommendations, and dissemination of results.
For each valid argument, participants received a reward. Participants could also challenge each other's arguments and get a reward for each valid counter-argument or condition. An argument was considered invalid if it had been mentioned before by someone else for the same research phase. The same arguments and conditions could apply to different research phases. The reward was a LEGO® block with which the group could build a construction after finalization of the game.
The game master supervised the flow of the game, made sure the rules were followed, made notes of the arguments, and monitored the time. After the game, each group selected its main argument pro, contra, and its main condition for PI, across all phases of policy research. These were brought together in a plenary session, where all groups presented their main arguments, followed by a plenary discussion with all participants. The plenary discussion revealed interesting insights into the prevailing culture about PI at KCE, irrespective of the research domain.
HTA researchers were active participants in the discussion and got feedback on arguments and statements from other researchers, which is demonstrative for the close interactions between research fields at KCE. Therefore, we report on all results that are applicable to HTA.
Development of draft position statements
In the next step, the draft position statements were developed. To nourish the position statements, besides the culture measurement, a scoping review of the literature on PI in health research was performed, PI initiatives in other countries and networks were described, Belgian researchers with experience with PI in research in Belgium and the three Belgian patient umbrella organizations were interviewed. The methods and results of the scoping review and interviews are described elsewhere (Reference Cleemput, Dauvrin, Kohn, Mistiaen, Christiaens and Léonard10). The interviews with the patient umbrella organizations provided information on their perspectives with respect to the feasibility of PI approaches (intensity and phases) in KCE research.
PI initiatives such as INVOLVE and the James Lind Alliance in the UK, HTAi, G-BA in Germany, Healthcare Improvement Scotland, the Canadian Agency for Drugs and Technologies in Health, EUnetHTA, RedETS, and some smaller initiatives were studied to exemplify how PI could be conceived in an HTA agency (Reference Cleemput, Dauvrin, Kohn, Mistiaen, Christiaens and Léonard10).
The co-authors of this manuscript formulated twenty-one draft position statements based on the input of the literature review, the interviews, and the arguments and conditions obtained from the nominal groups.
The support for the draft position statements amongst all KCE employees was assessed through a two-round face-to-face Delphi process, followed by an online survey to collect written comments on the draft statements. The participants to the Delphi panel first voted on each position statement (first Delphi round), using one out of four response categories: agree, almost agree (agree with the principle but reformulation required), disagree somewhat (tend to disagree, but no very strong feelings), strongly disagree. People who have no opinion on a particular statement were asked not to vote. Participants could express an opinion against PI in general by voting negatively on the first statement about KCE taking a positive position toward PI in health policy research. The votes were anonymous. Consensus for acceptance was defined as 75 percent or more of the respondents voted “agree or almost agree” AND less than 10 percent voted “strongly disagree,” consensus for rejection as 75 percent or more voted “disagree or disagree somewhat” and less than 10 percent voted “agree.” If consensus was reached, the statement was adopted. If no consensus was reached, the statement was submitted to a second voting round after all statements had been voted upon and a plenary discussion on that statement had taken place (second Delphi round).
After the second Delphi round, everyone had the opportunity to comment or make suggestions for all statements via an online questionnaire. Based on the comments of the meeting and in the web-survey, the research team discussed (1) for statements that reached consensus, whether re-wording would clarify statements without changing the basic idea of the statement; if so, a reformulation was proposed, and it was assumed that in these cases, consensus would still be reached for the clarified statement, and (Reference Berglas, Jutai, MacKean and Weeks2) for statements that did not reach consensus, whether a reformulation is possible that would increase the agreement amongst the KCE staff; this was done only when all comments for a particular statement went into the same direction. When the comments went in different directions, no changes were made to the statement. The final decision about the reformulated statements was made by the general management. Rejected position statements, or statements which could not be reformulated to meet the comments, were not retained.
Results
PI culture
The results of the nominal group of HTA researchers are presented in Table 1. The group started the “patients on board”-game at the “scoping”-phase. There was insufficient time to discuss PI in the dissemination and recommendations phase.
Table 1. Arguments pro and contra PI and conditions for PI in HTA
The raised arguments in favor of PI in HTA related to the increased relevance of the scope, expertise with data collection, potential to bring in fresh ideas for the study design, access to survey participants, validation of data analyses, better adherence to recommendations.
Disadvantages and risks mentioned by the participants included the lack of scientific knowledge of involved patients, credibility of the HTA, the added value of PI, required resources for PI, conflicts of interest, and heterogeneity within patient populations. It was also emphasized that PI should by no means replace scientific research on patient-related aspects. Concerns were raised about patients' subjectivity, emotions, and lack of ability to distinguish between their personal problems and the more macro-oriented perspective of HTA, thereby slowing down the HTA process. With respect to conflicts of interest, it was highlighted that some patient organizations are fully funded by the pharmaceutical industry, and might therefore have a conflict of interest when they contribute to an HTA on a particular pharmaceutical product or when they submit topic proposals.
Conditions for effective and meaningful PI in HTA related to measures that would mitigate the identified disadvantages, such as conflict of interest management, methods for identification and selection of patients to be involved, and careful consideration of whether PI is really expected to be valuable for the HTA before embarking upon the activity. HTA researchers also felt that it is important for KCE to develop clear methods and procedures for PI before starting a systematic approach, defining patients' role and expected input, and researchers' freedom to decide what to do with the input of patients.
Similar arguments were observed in the other nominal groups. Some arguments return in different phases of the research cycle, for example, potential conflicts of interest, time consumption, lack of knowledge.
During the plenary session, also phases that were not addressed in the HTA nominal group were discussed. For example, in the “call for proposals”-phase, the involvement of patients was considered to be beneficial to avoid that topics submitted relate mainly to highly prevalent conditions or very science-driven topics. However, it was also argued that increased PI in this phase might also reduce the overall appropriateness of the KCE research portfolio, because KCE has a societal remit, and also needs to tackle societal issues that surpass individual patients' concerns. Too much focus on individual patients' priorities might neglect other important healthcare topics with a strong societal impact. The current processes at KCE allow every citizen, organization, or institution to submit proposals. Some plead to keep it that way, and not put additional efforts in collecting proposals from patients. It was also mentioned that for the prioritization of assessment topics, a new selection criterion such as “relevance for patients,” should be included.
There was disagreement regarding the appropriateness of involving individual patients in the recommendations phase. Some argued that patients can bring in the daily life perspective, to make sure the recommendations following from the assessment make sense in practice. Others considered this inappropriate to involve patients in the recommendations phase because individual patients are rarely a direct target group of the recommendations. There was general agreement, though, that involvement of the patient umbrella organizations in the discussion on the recommendations at the Board of KCE is important. For the dissemination of results, it was mentioned that patients could help developing patient summaries or patient fiches.
Because of the concerns related to appropriateness, several conditions were identified for PI at KCE, such as “patients should only be involved if primary data collection in patients is needed,” “patients should not be involved if the study relies on quantitative data analysis only,” “PI in the analysis phase might be appropriate but should be a free choice of the research team.”
Finally, the issue of representativeness of patients involved was raised for several research phases. Heterogeneity within patient populations was considered as a barrier for PI in the call for proposals, the scoping of the research study and the design. However, if specific conditions with respect to representativeness could be met, the balance could move to more support for PI. It was considered important by the KCE members that it can be assured that patient representatives involved are representative of a sufficiently large patient population. For example, umbrella organizations of patients associations, representing a wide range of patient organizations, are currently involved in the selection of topics. They have acquired experiential knowledge for several conditions, by collecting experiences from a wide range of patient populations.
For the design phase, a possible condition could be to always include the same group of patient representatives, or to apply a selection procedure for patients similar to the selection procedure for subcontractors. These could be “expert”-patients, who are or became familiar with scientific approaches.
Endorsement of draft position statements on PI at KCE
The 21 draft position statements related to the fundamental ethical, instrumental, and procedural rationales for PI, the consideration of the relevance and need for PI, the complementarity of PI to patient-based evidence, required resources for PI, impact on planning, required training, evaluation, reporting, feedback, and who to involve, when, and for what purpose.
Nineteen statements were submitted to the Delphi panel. Two statements were not, because they reflected current, already widely accepted KCE work processes: (i) involvement of patients in the call for proposals and (ii) involvement of patients in the formulation of the recommendations, by means of the patient umbrella organizations.
Fourteen out of nineteen statements reached consensus for agreement and one for rejection in the first Delphi round (Table 2). After the discussion of the four statements for which there was dissensus in the first round, dissensus remained in the second round (Table 2). The arguments given during the plenary discussion after the first Delphi round are summarized in the last column of Table 2, together with the comments obtained from the online survey. Forty out of 71 KCE staff members responded to the survey.
Table 2. Draft position statements presented to KCE for voting, results first and second Delphi round, comments from plenary discussion and online survey and final decision
The comments during the discussion and on the web-survey provided important insights. First, they revealed that it was insufficiently clear to the respondents that the position statements relate primarily to PI for the purpose of better addressing patient issues in an assessment, and not to involving patients in all the decisions made during the assessment. Depending on the topic and the available patient-based evidence, it might be that PI is only required for specific assessment elements (e.g. patient-relevant outcomes) and not for others (e.g. cost-effectiveness analysis). Second, the use of the term “should” in the statements triggered resistance. It should be made clear that this term only implies an obligation in situations where PI is considered relevant because there are (potential) patient issues. Third, the specification of who should be involved as patient representative (individual patients, patient organizations, or others) in the position statements triggered many concerns and questions. It was decided to remove these specifications and use the term “patients” throughout the position statements as an overarching concept, encompassing different types of representatives. The specification of who should represent patients in which stage and for which purpose was out of scope of the current study but will be developed in a next stage in a practical process note.
After consideration and adaptation of some statements, eight out of the twenty-one draft statements were reformulated and three rejected. The rejection of some draft position statements was possibly in contradiction with what we know about the usefulness of PI. Regarding the statements for which there was dissensus in the second Delphi round, the following decisions were taken:
• A statement for which all the comments went into the same direction, for example, not agreeing with co-production of elements to be considered, but all agreeing with consultation, was retained after re-formulation.
• The statement about the consultation of patients to select and test data collection instruments was retained by the management.
• The heavily debated statement about the consultation of patients for the interpretation of the study results was rejected.
• The statement about giving the patients the opportunity to review the synthesis and give feedback before publication was rejected for two reasons: first, practical issues make this approach unfeasible. Belgium is a country with three official languages. It is not feasible nor efficient to translate the synthesis in three languages before review and to revise all three language versions after the review. This would take disproportionally much time and resources. Second, currently the patient umbrella organizations actually already review the synthesis before publication, as members of the KCE Board.
The two position statements that confirm and perpetuate an actual situation regarding the role of patients in the call for proposals and formulation of recommendations are included in the final statements to strengthen the position of KCE on these points.
The final position statements are presented in Box 1.
Box 1. Final position statements on PI at KCEFootnote 1
• KCE wants to involve patients as much as possible in its HTAs, in order to support choices to be made during the HTA process about the (best) ways to evaluate patient-related aspects. The KCE's position statements are inextricably linked, and must be considered as a whole.
• KCE perceives the fundamental ethical, as well as the instrumental and procedural rationales for PI decisive enough to take a positive position toward PI in HTA and other types of health policy research. Patients have the democratic right to be involved in research about them, and they can contribute a unique perspective to the research from their personal experience, competences and knowledge.
• KCE aims to involve patients in all HTA phases if this is relevant and appropriate for the assessment. Patients should not necessarily be involved in all HTA projects. The relevance and need for PI should be assessed project by project.
• PI is complementary to the review of scientific evidence and primary data collection, not a substitute for it.
• Sufficient resources (human, financial, time) should be made available to ensure and support effective PI. KCE aims to assure this availability.
• The planning and processes of the HTAs must be adapted to implement PI in an optimal way.
• Researchers and patients or patient organizations should be trained to effectively involve patients or be involved in HTA.
• PI activities in HTA should be regularly evaluated and procedures revised when appropriate.
• Patient contributions and their potential impact on the HTA process should be reported in the assessment report.
• Patients and KCE researchers should give feedback to each other about the collaboration, to potentially improve future collaboration.
• Everybody, hence also patients, can already today submit topic proposals to KCE. This possibility should be maintained.
• Patients should be consulted in the scoping of the HTA to allow assessors to better describe the context of the research topic, taking patient issues into account.
• Patients should be consulted in the scoping phase to define the patient-related elements that need to be addressed in the HTA.
• Patients should be consulted in the selection of the patient-relevant outcomes to be included in the HTA.
• Patients could contribute to the decision about the recruitment of study participants if primary data collection in patients or healthcare users is needed.
• Patients should be consulted in the selection and for the testing of the data collection instrument(s) to be used in patients or healthcare users.
• Patients should be consulted to define the minimal important difference in patient-relevant outcomes.
• Patients should be consulted to get input about the formulation of the policy recommendations. This is currently already the case, thanks to the presence of the Belgian umbrella organizations of patient associations in the Board of KCE. This possibility should be maintained.
• Patients should be invited to collaborate on the dissemination of the results of the KCE project.
Discussion
This study reports on the measurement of an organizational culture regarding PI in HTA and other similar types of health policy research. In general, although KCE staff was principally in favor of PI in their research and see some clear advantages, there was also some resistance. Researchers experienced in PI in their research tended to emphasize the advantages, whereas the less experienced staff members mainly raised concerns. To assess whether experience with PI changes the acceptance and rejection of statements, the PI culture will be re-measured in 3–4 years.
Main arguments in favor of PI were identified for the very early phases of an assessment (call, scoping) and for the last phases (recommendations and dissemination or results). In the early phases, it is considered that patients can bring in important knowledge on what is important to them, what their unmet needs are, and what their priorities are.
Main arguments against PI were situated in the middle phases of the research process: the design, data collection, and analysis phases. Patients' lack of knowledge on scientific approaches and potential conflicts of interest were mentioned as possible obstacles. Also the impact on the time needed for HTA was emphasized: having to explain all methodological choices to patients, in order to allow them to get ownership of the project, is considered too time consuming and hindering the research flow. This concern was reduced when it was explained that the scope of PI in HTA would be to improve the assessment of patient-related and patient-relevant issues in HTA, rather than to impact all HTA domains and hence patients should not necessarily be involved in the entire assessment process. In specific cases they could, but this would not be a requirement. For example, for the economic evaluation, the relevant input from patients should preferably come from patient-based evidence (data collected in patients). Patients do not have to be involved in methodological choices about the cost-effectiveness analysis. A learning from this observation of expressed concerns is that it is absolutely crucial to insist on the precise scope and expectations of PI for the project from the very beginning of the culture measurement.
We integrated different levels of involvement in our draft position statements, from targeted consultation to user-led involvement. The higher levels triggered much discussion amongst KCE researchers, who felt it would limit their autonomy. Consultation was easily accepted because KCE already consults systematically with stakeholders during research projects. INVOLVE advises against viewing these approaches as hierarchical levels, and stresses that the quality of the relationships built between patients and researchers, parity of participation, and impact of PI are more important and different approaches can be applied in one project (11;12). Main conditions for PI brought forward were spread across all phases of the HTA process. Conditions relate to disclosure of possible conflicts of interest, scientific knowledge and educational background of patients involved, clear procedures and processes for PI (who to involve, how to involve them), and availability of resources (time and budget).
By publishing position statements about PI and defining KCE's level of commitment toward PI, KCE expresses how it sees the role of patients in its HTAs and hopes thereby to create realistic expectations about PI in those assessments. The umbrella organizations of patient organizations were consulted by means of a semi-structured interview in order to take their position about feasible levels of involvement into account in the formulation of the draft position statements. Patient organizations are also represented in the Board of KCE that approved the position statements.
Important considerations for the endorsement of the position statements were that the management of KCE endorses the set of position statements and commits to freeing resources to allow for PI in HTA. Of course, KCE always has to find the balance between its commitments toward the patients and its legal remit. Moreover, KCE's resources are also limited, hence choices will have to be made and PI will not always be pursued if the expected added value is limited.
A general principle is that expectations from the involvement, both from the patients' side and the researchers' side, should be discussed openly at the beginning of the involvement process with all those involved.
The next step will be the development of a methodological guide with practical guidance for PI in HTA and other types of health policy research. This will cover several aspects, such as who to involve in which research phase, how to select the patient (representative) to be involved, and which method to use to guarantee meaningful PI.
Describing and understanding the PI culture of an organization is an important first step in the process of implementing PI processes. It allows to anticipate on perceived or expected disadvantages of PI, such as increased workload and time, loss of autonomy, biased assessments, and concerns such as conflicts of interests and lack of knowledge of patients and loss of credibility of assessors. Mobilizing examples of good experiences by colleagues who did already involve patients in their work might help to convince more skeptical colleagues. The measurement as such has already changed people's point of view.
When a process note for PI is developed, it will have to take the current culture into account, but should allow to deal with a changing culture as time goes by. With increasing experience, the guidance might change.
Conclusion
While overall, the position of KCE staff seems positive toward PI in HTA, there are also concerns and perceived risks and disadvantages that need to be mitigated by means of clear procedures and measures. Discussing the perceived and experienced pros and cons, as well as possible conditions for effective PI helps to change the culture. As such, culture measurement is in itself an intervention to make the organization ready for PI processes.
Acknowledgments
The research team would like to thank their colleagues—members of staff, experts, and management of KCE—for their support and participation during the development of the KCE position statements. The research team also thanks all the interviewees for the valuable input they provided for the development of the draft position statements.
Financial support
This research received no specific funding from any agency, commercial, or not-for-profit sectors.
