For over a decade, the international health technology assessment (HTA) community has been gradually expanding its efforts to involve “the public” in HTA and related policy processes. Rationales for bringing the public voice into the HTA process often cite the importance of making “more informed, transparent, and politically legitimate decisions” (1, p. 44) in a sector routinely faced with the competing pressures of rapid technology proliferation, ethical controversies, and resource constraints (Reference Gauvin, Abelson, Giacomini, Eyles and Lavis1–Reference Abelson, Giacomini and Lehoux3). Public involvement frameworks have provided helpful guidance and point to different models and mechanisms aligned with different goals for public involvement (Reference Abelson, Giacomini and Lehoux3–Reference Reuzel, vander Wilt, ten Have and de Bries Robbé5). Examples range from the direct representation of public or “lay” members on advisory or decision-making committees to more consultative approaches where citizens’ views are solicited through surveys and focus groups (Reference Moran and Davidson6–Reference Einsiedel and Ross8).
Recently, processes that emphasize deliberation and its hallmark features of informed, values-based reasoning (Reference Blacksher, Diebel, Forest, Goold and Abelson9) have been viewed as promising ways to elicit public values in the complex, contentious, and ethically controversial area of health technology policy. Experimentation with methods such as citizens’ juries have revealed both their popularity with citizens and their potential to inform a variety of HTA decision-making processes (Reference Menon and Stafinski10–Reference Bombard, Miller and Hayeems12). Yet evaluation of these models has typically focused on their procedural features (e.g., representativeness, inclusivity, access to adequate and appropriate information resources) (Reference Davies, Wetherell, Barnett, Littlejohns and Rawlins13–Reference Mitton, Smith, Peacock, Evoy and Abelson15) while paying less attention to assessing their contributions to HTA decision making or the links between the two.
Efforts to document the impacts of public involvement are challenged by conceptual ambiguity over terms such as use, impact and influence, and their different types (e.g., instrumental, conceptual, or symbolic) (Reference Weiss16) and measurement problems associated with linking “micro processes to macro impact” (11, p. 2). The impacts exerted by deliberative processes often play out over long time horizons and are opaque, even to those most actively involved in the decision-making process. However, a small number of ethnographic evaluations have sought to document both the types of impacts exerted by citizen deliberations and how these impacts are facilitated and constrained. For example, Jones and Einsiedel (Reference Jones and Einsiedel11) traced the “institutional learning” impacts of a national public consultation on xenotransplantation that used a citizens’ jury model in the early 2000s. Their results revealed that, despite the lack of substantive policy change, the exposure to innovative public engagement methods facilitated a “shift in institutional culture” reflected in a greater openness to including a broader set of actors in the technology policy process (11, p. 1).
Role clarity and the relationships between citizen deliberators and the expert processes to which they contribute also appear to shape the degree and manner in which deliberative outputs exert influence. In their evaluation of the early years of the Citizens Council of the National Institute for Clinical and Health Excellence (NICE), Davies et al. (Reference Davies, Wetherell and Barnett17) observed the Council's initial lack of “embeddedness” within the inner workings of NICE and the need to strike the right balance between maintaining the Council's independence from the sponsoring organization while ensuring that its outputs were on the organization's radar. Council members called for greater clarity about their role in relation to NICE decision making and for greater transparency about how their input would be used (18, p. 88).
Aside from these rare in-depth accounts, there have been few efforts to systematically trace the impacts of deliberative processes on the work of HTA organizations and their relevant advisory bodies. As a result, questions remain about how deliberative participatory structures inform the various stages of health technology decision-making processes. We seek to fill this knowledge gap by tracing the impacts of a deliberative participatory structure (the Citizens’ Reference Panel on Health Technologies [CRPHT]) established in Ontario, Canada on the deliberations of the arms-length advisory body (the Ontario Health Technology Advisory Committee [OHTAC]) and its government-supported secretariat (the Medical Advisory Secretariat, MAS). We then consider the design features that shaped these impacts and consider the prospects for using these types of deliberative structures to inform the activities of HTA organizations more broadly.
METHODS
Study Context and Background
The MAS and OHTAC have had a long-standing interest in stakeholder engagement and incorporating ethical and social values into their HTA processes (Reference Johnson, Sikich and Evans19). In 2007, a framework was developed which provided opportunities for public input at various stages in the HTA process (Table 2) (20). In 2008, through a university–government research collaboration, we initiated a study with two main objectives: (i) to establish a process through which Ontario citizens could inform OHTAC deliberations regarding the social values that should be considered in making recommendations for how health technologies should be used in the Ontario health system; (ii) to experiment with a particular method for engaging the public in the health technology policy analysis process which has been used to a limited extent in the HTA arena but more extensively in other health system contexts (Reference Blacksher, Diebel, Forest, Goold and Abelson9). The study was approved by the Hamilton Health Sciences/McMaster University Faculty of Health Sciences Research Ethics Board.
Citizens’ Reference Panel on Health Technologies (CRPHT): Key Features
Details pertaining to the recruitment and demographic composition of the panel, which are not the central focus of this study, are described elsewhere (Reference Bombard, Abelson, Simeonov and Gauvin21). The fourteen-member panel met on five separate occasions over 18 months to engage in informed, facilitated discussion and values elicitation through 1-day structured deliberation sessions. At each meeting, the Panel reviewed selected health technologies at various stages in the HTA process (Table 2). Five technologies were reviewed: (i) colorectal cancer (CRC) screening; (ii) percutaneous aortic value replacement (PAVR); (iii) breast cancer screening for average and high risk women; (iv) gene expression profiling (GEP); and (v) serologic testing for celiac disease.
Background materials were circulated in advance of each meeting, which included HTA evidence summaries and draft recommendations (for three technologies), relevant review articles and newspaper clippings, and a workbook, which summarized the key attributes of each technology and the discussion questions. Each meeting agenda included an overview of each discussion topic followed by a Q&A session and a combination of large (externally facilitated) and small (self-facilitated) group discussions with reporting back and thematic summarizing sessions.
Relationships and Accountabilities Between the CRPHT, MAS, and OHTAC
The Panel functioned in parallel to the sponsoring organization to ensure its independence and credibility (Reference Davies, Wetherell, Barnett, Littlejohns and Rawlins13). Direct interactions among OHTAC, MAS staff, and the CRPHT members included only periodic visits from the head of the Medical Advisory Secretariat, the Chair of OHTAC, and one of its members to explain the workings of MAS and OHTAC, to reinforce the Panel's relevance and to observe its deliberations. Following their final Panel meeting, CRPHT members were invited to debrief with OHTAC about their experience. Beyond these interactions, MAS staff, OHTAC, and CRPHT members had only episodic exposure to each other when the results of their respective deliberations were shared by the research team through presentations and summary reports at OHTAC and CRPHT meetings.
Data Collection and Analysis
Conceptualizing and Documenting Impact
We conceptualized “impact” broadly to refer to the process through which input obtained from the CRPHT was considered by OHTAC and its secretariat, and how this input went on to influence OHTAC deliberations and the health technology advisory process it oversees. Our analysis was not guided by a predetermined framework, although we were aware of different types of knowledge use (Reference Weiss16) and the challenges of linking public input to changes in policy decisions (Reference Jones and Einsiedel11). Using the methods of descriptive qualitative analysis (Reference Sandelowski22), we systematically documented the various types of impact through OHTAC's discussions of the CRPHT reports and the actions arising directly from these. Furthermore, we gathered perceptions of impact from CRPHT and MAS staff and OHTAC members (Table 1).
Table 1. Types and sources of Citizens’ Reference Panel impacts
A combination of document analysis and interviews were used to examine the observed and perceived impacts of the CRPHT on the HTA process. Table 1 describes the types of impacts that were documented along with their corresponding sources. The same data sources also provided the basis for our analysis of the factors that shaped these impacts, where qualitative description and constant comparison methods were used as the main analytic techniques (Reference Sandelowski22).
RESULTS
Our analysis revealed two types of observed and perceived impacts exerted by the Panel on the five technologies reviewed at two different stages in the HTA process: (i) raising awareness of social values and ethics in HTA and (ii) informing HTA recommendations. Design features including periodic opportunities for direct and brokered exchange between the Panel and the OHTAC, which fostered role clarification, accountability and trust building, facilitated these impacts.
CRPHT Impacts on the HTA Process (Observed and Perceived)
Table 2 displays each of the phases in the HTA process where public input was invited; the technologies reviewed by the CRPHT; the input provided to MAS and/or OHTAC; and the documented impacts of the CRPHT on the HTA process. Results demonstrate that CRPHT impacts could be traced for all of the technologies reviewed in at least one phase of the HTA process.
Table 2. Citizens’ Reference Panel Impacts on Ontario's Health Technology Assessment Process
OHTAC: Ontario Health Technology Advisory Committee.
CRPHT: Citizens’ Reference Panel on Health Technologies.
Figure 1 displays these results more conceptually, emphasizing two principal types of impacts. At a macro level, the Panel's input had the effect of raising awareness about a range of societal and ethical values relevant to all five technologies reviewed. In doing so, this input correspondingly reveals the important role that public engagement can play in informing the HTA process. At a micro level, the Panel's input went on to inform OHTAC recommendations for three of the technologies reviewed (GEP, serologic testing for celiac disease, and CRC screening), suggesting a continuum of impact across the HTA stages for some technologies. Additional detail about how these impacts were exerted is provided in the following sections.
Figure 1. Conceptual Framework of Citizens' Reference Panel Impacts.
Raising Awareness About Social Values and Ethics in HTA
As described in Table 2, the CRPHT's input on the GEP vignette did not initially exert any traceable impacts on the HTA process based on the documents reviewed. However, interviews with staff acknowledged the value of the Panel's more macro-level input, which raised relevant quality concerns and implementation issues related to GEP testing (personal communication). Similarly, although the CRPHT's input on the PAVR vignette was deferred, the Panel's ability to meaningfully discuss the “larger societal issues” associated with high-risk last-resort technologies was seen as a valuable contribution to OHTAC deliberations (OHTAC meeting minutes, September 2009). Indeed, the Panel was commended for “having the courage to struggle with questions that pushed beyond the mandate of MAS and OHTAC, such as what is the best use of society's resources” (OHTAC meeting field notes, July 2010). And finally, CRPHT deliberations about breast cancer screening for average- and high-risk women identified the need to reconcile conflicts between scientific evidence and public attitudes, leading OHTAC to consider the value of more frequent face-to-face interactions between the CRPHT and OHTAC to discuss these issues in more depth (OHTAC meeting minutes, January 2010).
The macro-level impacts described in the preceding paragraph were reinforced through interviews with CRPHT members who shared mixed perspectives about the Panel's impact on the HTA process. Hopefulness tinged with skepticism characterized some of these views. For example, one member believed the Panel had a “negligible” influence, but believed it should continue to push for the role to become something more substantial than it was (pm863). Another questioned the degree of impact the public can actually have in policy making, and suggested that the most important contribution of the CRPHT was to “increase the awareness at MAS and OHTAC about the importance of public values, and the importance of seeking the public's opinion through structures such as the CRPHT” (pm263). This view was echoed by another member who believed that the importance of the panel was to create a “template” of citizen involvement that could be improved upon in the future.
To me, it didn't matter how much we were getting through at this point
in time. To me it was about setting up a process that was replicable . . .
I do think there's value in having an existing template that's really
refined, tested and viable. And I think that's where the communication
between OHTAC, them saying what they want and is needed from us
is absolutely crucial. (pm655)
Informing HTA Recommendations
At the micro-level, more traceable impacts were observed, the most striking of which was for the very first group of technologies the CRPHT reviewed - screening methods for the early detection of colorectal cancers and polyps (CRC screening). In their deliberations on the draft recommendation for this suite of technologies, Panel members voiced concerns about the potential loss of choice and patient autonomy associated with a population-based CRC screening programs and the pressure they might be under to be screened to avoid conflicts with their healthcare providers (Reference Bombard, Abelson, Simeonov and Gauvin21). OHTAC's consideration of this input resulted in the addition of a new section to the final OHTAC recommendation for CRC screening modalities:
Ethical and societal perspectives
People who do not wish to undergo CRC screening should be able
to decline it without fear that this will affect their relationship with
their physician (23).
The direct uptake of the Panel's input into the CRC screening recommendation was referenced by several Panel members as the clearest example of how their input influenced MAS and OHTAC's work.
I don't really know how much we may have influenced OHTAC to
this point . . . I would like to think that we made a contribution.
Whether we have? I would say yes, we did – we did get some wording
in a recommendation changed. (pm263)
Moreover, a senior MAS official acknowledged at a subsequent OHTAC meeting that “his thinking had been changed by the panel's position that patients had the right to say no” (OHTAC meeting minutes, July 2010).
While the CRPHT deliberations on GEP initially exerted a more macro-level (awareness raising) impact, our document review reveals that the Panel's input was explicitly referenced in the final recommendation for GEP (Table 2). The nature of the reference, however, suggests a less direct and more symbolic role (i.e., an additional source of qualitative research on ethical and societal implications of GEP).
Similarly, the CRPHT was also cited as an input to the “assess[ment] of societal and ethical determinants” (Table 2) in OHTAC's final recommendation for serologic testing for celiac disease. Before this, however, a member of OHTAC described the more macro-level influence it exerted on OHTAC deliberations (Figure 1):
[It] provided a reference point as we attempted to identify and evaluate
the importance of issues emerging from the information we were gleaning
from [other sources] . . . [the] material helped strengthen the process and contributed to a level of confidence as we commented on societal and
ethical values relevant to the OHTAC initiative.
(OHTAC member, personal communication)
Factors that Influenced CRPHT Impacts
The observed and perceived impacts of the CRPHT tell only part of the story of the CRPHT's impacts on the HTA process. Our results also point to several factors that appear to have shaped the Panel's impact, including the nature of the interactions between the CRPHT and OHTAC, the clarity of their roles in relation to each other, and the political landscape in which they operated.
Building Accountability Through CRPHT-OHTAC Interactions
CRPHT members cited several examples of features built into the CRPHT structure that gave them confidence and tangible evidence that their input was being considered by OHTAC. For example, each CRPHT meeting began with a “reporting back” agenda item that updated the Panel on what was reported to OHTAC since the previous CRPHT meeting, how OHTAC had considered their input, and any actions arising from it. Panel members were also aware that the CRPHT was a regular item on monthly OHTAC meetings, which served as a constant reminder about the CRPHT's existence and gave Panel members the belief that “[OHTAC] was always willing to hear our feedback” (pm655).
A second critical design feature was the presence of senior MAS personnel and the Chair of OHTAC at the first and fourth Panel meetings. These interactions were viewed as “highlights” by Panel members and reinforced the belief that their input was valued. As one member described, this “hit home for the panel that OHTAC was interested and wanted more” (CRPHT member, personal communication).
The invitation extended to panel members to attend an OHTAC meeting at the end of the study period provided further evidence of the CRPHT's perceived value and illustrated the benefits of direct interaction. In his direct remarks to the Panel members, the Head of MAS complimented the Panel for its “level of engagement and honesty” and for input “that went against the grain of what OHTAC was thinking” in the case of the CRC screening (OHTAC meeting minutes, July 2010; OHTAC meeting field notes, July 2010). Panel members who attended this meeting were left with the sense that OHTAC demonstrated a level of commitment to incorporating the Panel's input into their work (pm362).
Centrality of Role Clarification in Facilitating Panel Impacts
The opportunities provided throughout the study for direct interaction between MAS, OHTAC, and the CRPHT were described as helping to “solidify[ing] roles”. More of this type of communication and interaction would have given the Panel an even clearer sense of direction, as described by this member:
A little more communication from them to us, as far as feedback about what
was useful for them, what wasn't useful, especially in these beginning phases. That would have let me know that we were working in tandem, and that we also then would have a chance to ask of them, or to give them some guidance around, ‘oh well you know what, we think this one area is really important, and so I think it would help solidify our role as well if we had a little bit more interaction. . . . it didn't have to be a lot, it could just be one of the OHTAC members calling in our meeting . . . we can ask, how can we refine it and make it more useful for them. (pm655)
Although the CRPHT was established with a clear purpose—to provide MAS and OHTAC with social values and ethics perspectives to inform their analyses and deliberations—OHTAC members presented conflicting views about the kinds of issues they thought the Panel should weigh in on and the boundaries for their deliberations. These included, for example, the relative emphasis to be given to (i) technology-specific vs. broader themes such as the weight given to different types of evidence; or (ii) more focused social values input related to the adoption of specific health technologies versus value for money and opportunity cost considerations (OHTAC meeting minutes, September 2009; January 2010).
Navigating the Political Waters of Expert Advisory Bodies
Despite the structures put in place to foster accountability and to support role clarification, the CRPHT was an experimental structure with a time-limited mandate. Demonstrating its “value added” to a provincial advisory body operating in a dynamic political landscape was challenging at times. For example, several days following the Panel's deliberations on GEP testing an article appeared in a major Canadian newspaper announcing the Ontario government's decision to fund the same test (Reference Priest24). CRPHT members raised concerns that their time had been wasted deliberating on a technology for which a funding decision had already been made (CRPHT meeting field notes, May 2010). According to one CRPHT member, the GEP funding decision “shook the whole belief a little bit” (pm654) and cast significant doubts on their ability to influence the HTA process (OHTAC meeting field notes, June 2010). The unpredictable relationship between the expert advisory body (OHTAC), and the government decisions it sought to inform, undermined the perceived value of the CRPHT among its members.
DISCUSSION
The establishment of the CRPHT in Ontario, Canada, adds to the growing interest and experimentation with public deliberation models as vehicles for directly involving citizens in HTA and related health technology policy advisory processes. Our findings push beyond traditional evaluations of these models’ procedural features by tracing Panel impacts on the health technology policy advisory process. They demonstrate that bodies like the CRPHT can inform and exert tangible impacts on these processes within the limits of the expert advisory processes they inform. Panel impacts were observed for all five technologies reviewed by the CRPHT and at both early and later stages of the HTA process. They were more traceable and pronounced for three of them (CRC screening, GEP, and serologic testing for celiac disease), where the CRPHT input is directly referenced in resultant recommendations. The vignette stage, where the parameters of the evaluation are still being defined, appeared to be less amenable to direct impacts than the recommendation stage, suggesting that different types of impacts are exerted across the HTA continuum. These findings build on previous studies, which have evaluated the influence of public involvement on the HTA process (Reference Oliver, Armes and Gyte25).
Our findings also shed light on the factors that shape these impacts such as the interactions between citizen deliberators and expert advisory committees, the clarity of citizen roles in relation to the HTA advisory process, and the tenuous nature of citizen-expert relationships, which are embedded within broader political processes. Our findings add important new insights about one of the main design concerns of deliberative participatory structures—how to maintain the independence and credibility of the citizens’ panel while ensuring that it contributes substantively. Our results demonstrate that while precautions must be taken to minimize threats to the independence of citizen deliberators, these risks must be carefully weighed against the risks of citizens’ panels playing a “token” role or even being ignored by the expert body to which they report. Our findings demonstrate that both parties—the CRPHT and OHTAC—benefitted substantively from their interactions with one another, which in turn, facilitated CRPHT impacts on the HTA process. These interactions, along with the carefully brokered “report back” features built into CRPHT and OHTAC meeting agendas, appear to have mitigated both types of risks. Additional efforts could be taken to creating what Davies et al. (Reference Davies, Wetherell, Barnett, Littlejohns and Rawlins13) refer to as an adequate “expertise space” that values citizens’ lay knowledge so they believe they can contribute in a significant way to highly technical debates. This is achieved by framing issues to focus the deliberation on social and ethical dilemmas (instead of technical issues), providing easily-accessible background information, and an inclusive style of facilitation that highlights the original contributions citizens are able to make based on their values and experiential knowledge.
Finally, our results call for HTA and health technology advisory bodies to respond to the consistent messages from the public engagement literature for the sponsors and designers of public engagement to articulate clear and achievable goals for citizen deliberations in relation to existing organizational and decision-making structures (Reference Abelson, Giacomini and Lehoux3;Reference Pivik, Rode and Ward4). When the CRPHT was established in 2008, it was aligned with a decision-making framework that was early in its adoption and had not been rigorously tested with respect to the incorporation of social values into its decision making. As the study unfolded, competing visions of the role that social values should play and how the CRPHT might realize these different visions were revealed, but never fully resolved. As Davies et al. (Reference Davies, Wetherell, Barnett, Littlejohns and Rawlins13) observed of the NICE Citizens Council, deliberation takes place within organizational, policy and political contexts where power relations and accountabilities have already been established. In order for “new” deliberative structures to be productive, their roles in relation to existing decision-making structures must be clearly articulated, but also reviewed and re-negotiated to reflect evolving visions and mandates.
CONCLUSIONS
The establishment of the CRPHT as part of a collaborative research study provided a unique laboratory for experimenting with a deliberative participatory structure in the HTA field. It demonstrated that these types of structures can usefully inform different stages in the HTA process. Additional studies to build on these early observations are welcome, and should address important questions such as who the most appropriate audiences are for the deliberation outputs produced at different stages of the HTA process, and how this output should be synthesized and reported to most effectively inform the HTA process. Ensuring these structures are designed with opportunities for periodic exchange between citizens and experts to clarify roles, promote accountability and build trust will facilitate their impact in a variety of HTA settings.
CONTACT INFORMATION
Julia Abelson, PhD Professor, Department of Clinical Epidemiology & Biostatistics, Centre for Health Economics and Policy Analysis, McMaster University, Hamilton, Ontario, Canada
Yvonne Bombard, B.Sc., PhD Postdoctoral Fellow Yale University, School of Public Health, New Haven, CT, USA; Memorial Sloan-Kettering Cancer Center Center for Health Policy & Outcomes Clinical Genetics Service, New York, USA
François-Pierre Gauvin, PhD Lead, Evidence Synthesis & Francophone Outreach, McMaster Health Forum, Hamilton, Canada
Dorina Simeonov, BHSc, MSc Policy Analyst, Health Council of Canada, Toronto, Ontario, Canada
Sarah Boesveld, PhD Candidate Centre for Health Economics and Policy Analysis McMaster University, Hamilton, Ontario, Canada
CONFLICTS OF INTEREST
Julia Abelson received a research grant to her institution from the Ministry of Health and Long-Term Care (Ontario, Canada) to carry out the research reported on in this article. Yvonne Bombard received travel support from the same source. The other authors report they have no potential conflicts of interest.
