Health technology assessment (HTA) is a multidisciplinary process that summarizes information about the medical, social, economic and ethical issues related to the use of a health technology in a systematic, transparent, unbiased, robust manner with a view to informing decision making (1). HTA affects a wide range of stakeholders, and each has their own perspective that influences their opinion and how they view the HTA (Reference O'Donnell, Pham, Pashos, Miller and Smith2). Efficient use of resources and opportunity cost are important considerations in HTA, and the value of HTA is linked to the impact of HTA, and the extent to which it succeeds in informing decision making. Measures to maximize the impact of HTA include stakeholder engagement, quality assured processes and the structure, format and readability of the HTA report.
The purpose of this article is to highlight the contribution of stakeholder engagement to the impact of HTA. To illustrate the points, we present a case study of a national HTA undertaken in Ireland of a national public access defibrillator program.
BACKGROUND
Cardiac arrest is a sudden loss of heart function due to a malfunction of the electrical system of the heart usually due to an abnormal heart rhythm. Death occurs within minutes of the heart stopping. A cardiac arrest may be reversed by timely cardiopulmonary resuscitation (CPR) or use of an automated external defibrillator (AED) to deliver an electric shock through the chest wall and restore a normal heart beat. Public access defibrillation (PAD) programs are designed to improve survival from out-of-hospital cardiac arrest (OHCA) by reducing time to defibrillation before arrival of the Emergency Medical Services (EMS).
In 2013, The Public Health (Availability of Defibrillators) Bill was introduced in Seanad Eireann, the Upper House of Parliament in Ireland. This draft legislation Bill, which set out the specifications for a national PAD program, was sponsored by several independent Senators (3). The Bill proposed a list of premises and venues that would be required to install and maintain an AED and provide training to employees on its use.
The Bill was debated in the Seanad and received widespread cross-party endorsement proceeding to the Second Stage in development of legislation. The Seanad debate discussed experience from other jurisdictions and cited the potential that the program as described was likely cost-effective and could potentially achieve a survival rate of up to 40 percent (4). That expectation was partially based on the assumption that OHCA survival rates achieved at Dublin airport could apply nationally—an unlikely scenario in a country with a dispersed rural population. The Minister for Health stated that he would request the Health Information and Quality Authority (HIQA) to conduct a health technology assessment to inform the design and implementation of a national PAD program and the necessary underpinning legislation (4). The high level of public interest was evidenced by the extensive media coverage on the topic.
The HIQA conducts HTAs to inform national health policy and national health service decisions at the request of the Minister for Health and the Health Service Executive (publicly funded healthcare provider). The HTA is independent advice to inform a decision on investment or disinvestment and there may be valid reasons why a decision maker may consider other factors such as affordability, coherence with overall health policy direction, and political issues, such as the will of the people to fund an intervention despite it not being cost-effective.
METHODS
Synthesis of HTA Findings
The terms of reference for the HTA were developed in conjunction with the decision-maker's representatives in the Department of Health to include the following domains: Description of the Technology, Epidemiology and Service Configuration, Clinical Effectiveness and Safety, Cost-effectiveness, Budget Impact, Organizational, Social, Ethical, and Legal issues. The HTA was conducted according to our standard methods (5–9).
Stakeholder Engagement
Stakeholder engagement followed the standard approach as outlined in the relevant National Guideline (10). An Expert Advisory Group was established to guide and inform the HTA. The membership of the Expert Advisory Group included nominees of a broad range of stakeholders representing the decision maker (Minister for Health), providers (publicly funded health service, national ambulance service), clinicians (emergency medicine, cardiology, public health), first responders, cardiac patient representatives, the national out-of-hospital cardiac arrest register, international expertise in health economics, and in the design and implementation of PAD programs in the United Kingdom.
The role of the Expert Advisory Group was to inform and guide the process. Conflict of interest declarations were completed in line with HIQA policy. The Expert Advisory Group met on three occasions. Individual members were consulted outside of meetings on areas relevant to their own expertise and provided access to data and expert opinion to guide interpretation of data and assumptions in the economic model. The draft report was reviewed by the Expert Advisory Group before publication for consultation. A press release was issued by HIQA announcing that the HTA was under way, outlining the terms of reference and the membership of the Expert Advisory Group.
A public consultation on the draft report was conducted which included a targeted call to sixty-nine organizations. HIQA conducted a media campaign to raise awareness of the findings and the public consultation. Several interviews were conducted in the medical and national print media as well as on national and local radio. The consultation was also promoted on social media. Submissions to the consultation were analyzed, changes made to the report as required and a statement of outcomes itemizing all submissions and HIQA's response to each was published. The final report was reviewed by the Expert Advisory Group and approved by the Board of HIQA (11). HIQA met with the Senator who was the lead sponsor of the Bill on several occasions to brief him on the HTA process and what would be considered in the HTA. Immediately before publication of the consultation document, he was provided with a copy of the draft report and a comprehensive briefing on the findings.
The report was submitted to the Minister for Health as advice and published on the HIQA Web site, together with a press release and the statement of outcomes from the consultation. The report comprised the technical analysis, an executive summary, and a section summarizing the advice and underpinning findings. An infographic (Figure 1) was developed and published by means of the Web site and social media to highlight the key findings. Managed media coverage of the report included several radio and press interviews to raise awareness of the findings.
Figure 1. Infographic for HTA of national public access defibrillation program in Ireland.
An evaluation of the impact of the HTA was conducted by HIQA, which included relevance and value of activity, communication of the outputs, and adherence to internal processes.
RESULTS
HTA Findings
The HTA reported that there are approximately 1,850 OHCAs in Ireland every year, 76 percent of which occur at home. The survival from OHCA is 5.8 percent, which is lower than the international average of 7.1 percent. Approximately 24 percent of the Irish population have had CPR training, and 45 percent of OHCAs receive bystander CPR before arrival of the emergency response services. In the absence of a formal PAD program there has been widespread voluntary dispersal of approximately 8,000 to 10,000 functional AEDs in Ireland many of which are installed in locations with the highest incidence of OHCAs such as transport hubs. Implementation of the Bill would require provision of an additional 38,000 AEDs (11).
The widespread dispersal of AEDs in Ireland has been achieved on a voluntary basis and provides a similar AED density to countries with organized PAD programs (11). The widespread purchase of AEDs by voluntary organizations reflects the high degree of public support for publicly accessible AEDs, particularly in rural populations. However, dispersal of AEDs is not standardized, coordinated, maintained, or linked to emergency medical services (11).
Alternative PAD programs modeled in the assessment ranged from the comprehensive scheme outlined in the Bill to a scaled-back version involving only those building types with the highest incidence of OHCA. The HTA found that implementation of PAD was estimated to result in between an additional two to ten OHCA patients surviving to hospital discharge annually depending on the program implemented. The incremental cost-effectiveness ratios comparing each PAD program to the next most effective ranged from €96,000 per quality-adjusted life-year (QALY) for the most cost-effective program to €928,000 per QALY for the program described in the Bill (Reference Moran, Teljeur, Masterson, Harrington and Ryan12). The average cost-effectiveness ratio comparing the Bill with the current situation was €301,000 per QALY. PAD as outlined in the Bill would cost €105 million over 5 years, with the majority of that cost falling on small- and medium-sized businesses (11).
Scenario analysis indicated that, if use of AEDs by bystanders increased by approximately 40 percent, then the most scaled back option could be cost-effective (11).
Stakeholder Engagement
Twenty responses to the public consultation were received comprising over 100 comments, which were analyzed and processed. Half the submissions were made in a personal capacity, 30 percent by industry, and the balance by healthcare organizations. The statement of outcomes published with the final HTA report outlined the submissions and the HIQA response (13). The report was updated to include some additional information and to clarify certain aspects of the conclusions, findings, underlying assumptions, and the methodological approaches chosen.
The sponsoring Senator welcomed the announcement that the HTA was under way during a speech in the Seanad and outlined the scope of the HTA (14). Later, he and other members of the parliament encouraged participation in the public consultation (15;16). When the final report was published, the Senator stated that he accepted the report's findings (17).
Positive media management to encourage uptake of the story resulted in substantial coverage by national, medical, and local print, on-line, and radio media as recorded in the impact assessment.
A decision was made not to progress with the Bill. In the responses to the questionnaire on the impact of the HTA, the decision-maker's representative highlighted the importance of the HTA in informing the decision particularly given the prior dearth of relevant Irish data.
DISCUSSION
HTA Findings
The HTA quantified the clinical benefits of the proposed PAD program as modest, identified that substantial costs would fall on small and medium businesses at a time of economic recession, and that none of the proposed programs could be considered cost-effective. The HTA advised that, if a PAD program is introduced in Ireland, it should be considered in conjunction with measures to improve cost-effectiveness, such as increased use of publicly accessible AEDs, by increasing public awareness, expanding CPR and AED training, and establishing an EMS linked register. Any prospective program should start by targeting the mandatory deployment of AEDs to locations with the highest incidences of OHCA. A process of performance evaluation and research should be incorporated from the outset to guide ongoing tailoring of the program to maximize its efficiency (11).
While the Bill did not progress, there is now a policy imperative to strengthen the voluntary First Responder network, establish a register of AEDs linked to EMS, and promote use of AEDs by means of support for CPR/AED training for citizens and increased public awareness.
Stakeholder Engagement
Ensuring appropriate stakeholder engagement in the development of a HTA is important to the integrity of the process, the quality of the outputs, and acceptance of the findings (Reference Teljeur, Harrington, Moran and Ryan18). Stakeholder engagement facilitates the generation of relevant, transparent, and effective advice providing quality assurance of both the process and content of the HTA. It also allows the creation of a shared understanding and acceptance of the findings. These objectives are achieved through the standard activities of engagement with an expert advisory group comprising stakeholder representatives, public, and targeted consultation, and positive media engagement. In Ireland, HTA recommendations constitute advice rather than binding guidance. It is important for the transparency of decision making that the HTA findings and underpinning evidence are comprehensive and clearly presented.
This HTA was unusual in terms of its direct link to development of legislation and the high level of public awareness of the issue, but the approach taken was consistent with HIQA's standard stakeholder engagement principles of providing accessible information in an open, transparent manner while ensuring the integrity and independence of the HTA process (10). It was, therefore, important to ensure that the political stakeholders and members of the public were appropriately briefed on the HTA and its findings. Political interest in this HTA was managed through providing the lead sponsor of the Bill with an explanation of the HTA process at initiation of the HTA and subsequently an understanding of the key findings that underpinned the advice. Care was taken to explain the robust processes undertaken to generate the objective scientific evidence, thereby conveying confidence in the strength of the findings underpinning the advice.
Nominees from key stakeholders were represented on the Expert Advisory Group ensuring that they had sight of the data and understood the generation of evidence that was the basis of the advice. Making the report available to targeted and public consultation further provided stakeholders and the public with an opportunity to provide comment and feedback before the report being finalized. Media management was used to increase public awareness of the consultation. The Senator who had proposed the Bill and colleagues made statements in support of engagement in the consultation process.
Press releases are used routinely by HIQA to ensure that the important findings are appropriately and accurately summarized in user-friendly language. Press releases generally take the form of one page of statements and quotes, followed by several facts relating to the burden of disease, performance of the intervention, and other relevant contextual information. An HTA often involves complex concepts expressed in technical language that is not readily accessible. Providing a brief summary assists the media to identify the key messages in the HTA. Positively managing the media coverage is important to ensure that the study findings are clearly and unambiguously presented, thereby minimizing the risk of inaccurate and misleading commentary. Proactive media management also promotes dissemination of the findings, which is important to maximize public awareness.
An evaluation of the impact of the HTA included a questionnaire issued to members of the Expert Advisory Group and to the Minister's representative from the Department of Health (Reference Teljeur, Harrington, Moran and Ryan18). Stakeholder perception is important irrespective of the scientific quality; a poor perception is likely to diminish impact. The Expert Advisory Group members were positive about the findings of the report, and believed that it was a fair interpretation of the available evidence. Decision-maker confidence in the scientific reliability and accuracy of a HTA has previously been described as an important contribution to its impact (Reference Rosen and Werko19). Follow-up with the decision-maker's representative confirmed his confidence in the report findings.
The impact of the work of the Austrian HTA agency was previously evaluated in terms of its use in decision making and the consequences for healthcare expenditure and resource distribution (Reference Zechmeister and Schumacher20). A key finding was the need to systematically incorporate HTA into decision making to increase impact. This HTA was specifically requested by the decision maker to inform an imminent decision on progression of proposed legislation thereby increasing the likelihood that its findings would be considered in the policy-making process.
The case study used to illustrate this article was subject to several limitations. The impact of a HTA is related to a variety of factors, including the scientific integrity of the work, the context for the HTA topic and associated decision-making process, and the extent to which stakeholders are involved in the process and accept the findings of the HTA. Measuring the relative contributions of the many factors for a given HTA is highly complex. It must also be borne in mind that there may be a substantial lag between publication of a HTA, and the implementation decision that follows it.
In the case of legislation, which can be subject to a lengthy development process and can be redrafted, the intervention that is finally adopted may bear little relation to that originally proposed and evaluated in the HTA. The impact assessment that provided learning to improve the impact of HTAs included quantification of the media coverage, compliance with the quality assurance framework, and assessment of perception of the quality and utility of the HTA. We used questionnaires to gather feedback from key stakeholders involved in the process and identify what they perceived as contributing to the impact of the HTA. Impact was measured ultimately in how the HTA influenced the decision-making process. The HTA showed that the proposed intervention was not cost-effective and would have a modest impact on survival from OHCA. After publication, the findings of the HTA were accepted by the decision-maker's representatives (Department of Health) and by the politicians who sponsored the draft legislation. Therefore, on the basis of the findings and advice in the HTA, the Bill did not progress.
CONCLUSION
Over time, HIQA has developed several strategies to maximize the impact of HTA (see Figure 2). Key stakeholders for this HTA included members of parliament and the general public. Stakeholder engagement for this HTA was tailored to ensure that these key stakeholders were informed of the HTA process, the generation of evidence, and the advice thereby maximizing its acceptance.
Figure 2. Stakeholder engagement steps to maximize the impact of HTA.
This HTA was unprecedented in that it was requested to directly inform development of legislation and that the topic was already the subject of substantial discussion in the parliament and among the public. The HTA provided the evidence to inform the decision not to progress with legislation for a national public access defibrillation program. This demonstrates the importance of timely evidence synthesis in the development of public health legislation that has considerable implications for a broad range of stakeholders. In the absence of this analysis, there is a danger that well-intentioned initiatives may be pursued, despite being associated with prohibitive opportunity costs that may not be readily apparent during the legislative process. Once enshrined in legislation, there would be considerable difficulty in withdrawing these interventions when the full extent of the opportunity cost became apparent.
One-to-one interaction with key stakeholders, engagement with the Expert Advisory Group, use of public and targeted consultation, and positive media management facilitated acceptance of the message among those responsible for implementation and support of uptake of the HTA advice. Appropriately tailored stakeholder engagement is integral to the impact of HTA.
CONFLICTS OF INTEREST
The authors have nothing to disclose.
