The European Network for Health Technology Assessment (EUnetHTA) (www.eunethta.eu) was established in 2005 to create a network for health technology assessment (HTA) collaboration across Europe. Activities have been delivered through a series of projects and Joint Actions (Reference Kristensen1). Joint Action 2, which ran from 2012 to 2015, aimed to strengthen cross-border HTA collaboration through development and testing of practical tools and approaches. Work Package 5 (WP5) within Joint Action 2 tested the feasibility of national HTA bodies collaboratively producing structured rapid HTA reviews, according to an HTA Core Model for rapid Relative Effectiveness Assessment (REA), and applying it in national contexts (http://www.eunethta.eu/tools).
The Scottish Health Technologies Group (SHTG), part of Healthcare Improvement Scotland (HIS), acted as a collaborating partner in this work which was led by the National Healthcare Institute in the Netherlands and the Ludwig Boltzmann Institute for Health Technology Assessment in Austria. SHTG participated in the capacity of “dedicated reviewer,” in the production of three EUnetHTA rapid reviews. This entailed providing peer review at all stages of the work from protocol generation to the finished product. The topics of these rapid reviews were: renal denervation systems for treatment-resistant hypertension (2); transcatheter implantable devices for mitral valve repair in adults with chronic mitral valve regurgitation (3); and endovascular therapy using mechanical thrombectomy devices for acute ischemic stroke (4).
This study describes our experiences of adapting two of these EUnetHTA rapid reviews (3;4) to inform SHTG advice for the National Health Service in Scotland (NHSScotland). It outlines the approach followed, discusses lessons learned, and issues for other agencies to consider when making such adaptations.
SHTG prepares rapid reviews called Evidence Notes using a process accredited by the National Institute for Health and Care Excellence (NICE). These describe the health technology in question and the epidemiology of the indication, and then summarize the clinical effectiveness, cost effectiveness and safety evidence. Evidence Notes are then used by the SHTG to develop Advice Statements. These briefly summarize the evidence on the clinical and cost effectiveness and safety of a healthcare technology alongside contextual information. They conclude with advice around the implications for NHS- Scotland for consideration by healthcare decision makers. As an organization always striving to improve its efficiency, HIS was interested from the outset of the EUnetHTA project, in evaluating how the EUnetHTA joint work could be adapted into Evidence Notes and subsequently used to develop advice for NHSScotland. Questions of whether this could save time and resource whilst providing a product at least as robust and relevant as if it had been entirely developed locally were of foremost interest.
The SHTG had already been working on the topic of renal denervation for some time when this European project was proposed so there was limited scope for using any of the European work to inform the Scottish advice. However, SHTG were able to contribute the knowledge of the topic and literature already gained to the EUnetHTA report by acting as peer reviewers on this topic, and we noted our review on this topic in the EUnetHTA Planned and Ongoing Projects database to raise awareness of our work. When the second topic, on mitral valve repair, was proposed, SHTG consulted with clinical experts in Scotland who indicated that this topic was of interest within NHSScotland and that an adaptation of the EUnetHTA work to produce an Evidence Note and advice would be useful and timely. One of the Scottish experts agreed to act as a clinical reviewer for the European group, further strengthening our involvement with this project and potential for adapting the work for Scotland. The third topic proposal, on mechanical thrombectomy, was notified by EUnetHTA at the same time as a similar request for an Evidence Note was received by the SHTG. This request indicated that a very small number of procedures were currently being undertaken on an ad hoc basis in Scotland, but that information was needed to assess the potential merit in scaling up the provision.
METHODS
Working with the EUnetHTA Core Model for REA
The HTA Core Model for REA is based upon the EUnetHTA Core Model for full HTAs, which defines the content of an HTA as a series of domains, topics within the domains, and issues within topics, that translate into the review questions. This modular approach aims to facilitate the process of co-production of HTAs and their subsequent adaptation within national contexts. The nine domains in the full core model are reduced to four in the core model for REA. These four domains are: (a) description and technical characteristics of the technology; (b) problem and current use of the technology; (c) safety; and (d) clinical effectiveness. The EUnetHTA rapid reviews do not presently cover health economics.
Initial Assessment of Relevance and Feasibility of Adaptation
For both Evidence Note projects, we had contact with local NHSScotland clinicians who were able to advise on relevance of the topic and who also acted as clinical experts for the European reviews. This increased our confidence that the review questions considered in the European reviews were aligned with the needs of NHSScotland. For the Evidence Note on mitral valve repair, following clinical consultation, we only included the evidence on one of the three technologies considered within the EUnetHTA rapid review. The other two technologies were considered to be at too early a stage of development to be of immediate interest in Scotland.
To accompany their core model, EUNetHTA developed an adaptation toolkit to enable agencies to systematically assess the feasibility of adapting an HTA product. For both topics, we used the screening questions set out within the adaptation toolkit to confirm that no methodological, language or conflicts of interest issues were likely to be a barrier to proceeding with adaptation. It was clear from the outset that because EUnetHTA rapid reviews do not include health economics, any adaptations would need to be supplemented by an in-house review of the economic evidence. Consideration of cost-effectiveness is a key aspect of providing advice for NHSScotland.
Adaptation Process
Having made this assessment of relevance and feasibility we then set about summarizing the report to enable it to fit with the format of our own Evidence Notes. The concise, structured format of the Evidence Notes is recognized and liked by our local decision makers in Scotland. We also undertook several additional pieces of work, and these are described subsequently with a view to sharing our reflections on the merits or otherwise of these.
Additional Literature Searches
We undertook our own literature searches to identify published economic analyses, for both topics. Given that we were doing this, we took a decision to also search to check if any additional systematic reviews had been published since the completion of the EUnetHTA literature searches. For the thrombectomy topic, we updated the long list of ongoing studies which was presented in the EUnetHTA report.
Supplementing Epidemiological Information
For the mitral valve repair device, we sought data on the target population size, as EUnetHTA had been unable to identify such data at a European level. For the mechanical thrombectomy review, we chose to expand upon the information given to provide Scottish epidemiological information, especially as stroke is a condition for which Scotland has detailed high quality data. While EUnetHTA reports could provide detailed country level epidemiological information, we think that this is more sensibly done at a local level by individuals with knowledge of and recourse to local sources.
Additional Device Information
We searched for additional information to try to more closely specify the indication for the mitral valve repair technology as this was very broad within the EUnetHTA report. As much detail as possible is required on the device indication to aid adaptation. We also sought for both our reviews, descriptions of the technology to provide an alternative to referencing the manufacturer submissions that formed part of the EunetHTA rapid review report. We aimed to identify openly available sources only.
RESULTS
For both Evidence Notes, replicating the search strategies published by EUnetHTA was not as straightforward as we had hoped, although it was easier and less time consuming than designing new searches. It is rarely possible to copy search strategies directly, and there is always a need to adapt searches based on differences in shorthand and terminology of different interfaces. Liaison with the EUnetHTA team provided us with their bibliographic database files, which did save on administrative work.
Mitral Valve Repair
Economic analyses identified for the mitral valve repair device, focused on data from a study which was excluded from the clinical effectiveness considerations in the EUnetHTA review on the basis of its comparator group. We considered it necessary to report this study in the clinical effectiveness section of our Evidence Note due to the prominence of its use in economic analyses. However, this then led to our Evidence Note diverging from the EUnetHTA inclusion criteria.
Six new systematic reviews were identified from our update searches, one of which we incorporated into our Evidence Note. The additional review compared outcomes of medical therapies for mitral regurgitation, using historical data, with contemporary outcomes for our device of interest. Because no head to head comparisons between this device and medical therapies in the patient group of interest had been identified by the EUnetHTA review, we believed that this systematic review, while it had methodological shortcomings, had potential to inform discussion on the topic.
Exploring further device indications for the mitral valve repair topic, brought little additional value in terms of elucidating the EUnetHTA statement that the indication for the device in Europe is much broader than in the United States.
Endovascular Therapy with Mechanical Thrombectomy
A large number of new systematic reviews had been published since the completion of the EUnetHTA literature review. There were ten new systematic reviews which met the inclusion criteria. Our intention when deciding to include these was mainly to extract findings from subgroup analyses likely to be of interest to decision makers in NHSScotland, but also to assess consistency of findings across systematic reviews with different inclusion criteria, particularly inclusion of unpublished studies.
CONCLUSIONS
Overall, both the mitral valve repair (5) and the mechanical thrombectomy (6) Evidence Notes were developed well within usual timescales for production and there were no notable differences identified in process, or engagement, of using these adapted evidence reviews to form advice for NHSScotland compared with Evidence Notes that we have developed from scratch ourselves. However, having done these initial two adaptations we can make several observations about lessons we have learnt for future such work.
Timing
Both of these adaptations were for technologies other than medicines. Timing for the assessment of nonmedicine technologies is generally less critical than for medicines, certainly in the United Kingdom. However, it is still necessary for European work to be available at appropriate times for local adaptation to be useful. In both these cases, timing was suitable. However, we did struggle to identify and engage with experts in the short time scale between becoming aware of European work going to be undertaken and its commencement. Ideally a long-term forward work plan would be available enabling local agencies to plan when and whether adaptations may be an option for them.
Summarizing
EUnetHTA rapid reviews are longer than our standard Evidence Note format and do not cover health economics, so the possibility of taking the full report, or the executive summary and re-badging it, while it would have been time saving, was immediately not an option for us. Given the difference in lengths and detail, judgement was required throughout the adaptation process on what information to include in our Evidence Note from the EUnetHTA report and how best to present this. For example, time frames and resources would normally limit us to selecting and working with only large observational series for the safety section of our Evidence Notes. The EUnetHTA mitral repair review included two large registries and also thirteen smaller studies.
We decided in our adaptation to report all safety data as in the EUnetHTA review because it was easily available, well presented, and offered readers the benefit of this additional information. While a shorter EUnetHTA review may have been more easily translatable to our format, the modular structure of the review did make the extraction of sections straightforward, and also given that the EUnetHTA reviews are intended to be adapted across Europe, the format could never be tailored to suit all. Also having such a detailed report, negated the need to seek out source literature, which may have been the case had the original EUnetHTA reports been shorter. Simplifications to the rapid review report structure took place during the course of WP5 resulting in outputs that were clearer and less cluttered for readers and users of the reports.
Cost-Effectiveness Analyses
There are arguments that it is inappropriate to include economic analyses within EUnetHTA reports as they need to be designed for the local context. However, we consider that the summarizing of appraisal of economic analyses from different contexts is still helpful as the closeness of fit to the local decision-making context can then be assessed and also a general indication of the magnitude and direction of cost per QALY ratios can be obtained. Inclusion of this component of HTA within EUnetHTA reviews would have assisted with our local decision-making process. If economic studies were to be included, providing detailed information on the decision-making context considered within them would assist in assessing their relevance for local adaptations.
Supplementing and Updating the EUnetHTA Review
We put considerable effort into ensuring that our adapted Evidence Note was up to date and complete. However, in retrospect, it is doubtful as to whether the work of updating the list of ongoing trials for the mechanical thrombectomy topic added value in our context. For a technology with no randomized controlled trial (RCT) evidence, the information that there are trials in progress or soon to publish is valuable. However, for topics with an existing body of RCTs such as this one on mechanical thrombectomy, and many studies ongoing, expanding the list to cover a few months of additional information is unlikely to contribute to decision making and simply adds on extra time.
In terms of supplementary literature searches, it is questionable for the mitral valve repair Evidence Note whether the additional effort in searching, sifting, and appraising the additional studies added value to the review over and above the EUnetHTA work, because the additional paper only confirmed the need for robust contemporary studies. For mechanical thrombectomy, the majority of the additional systematic reviews identified were based upon the same primary data as that incorporated in the EUnetHTA review, and thus the added benefit of including them is debatable. Having a central repository for each EUnetHTA report where additional studies identified subsequent to publication could be highlighted such that other agencies could access this information for any future work they may carry out, could be an option for EUnetHTA to consider for future.
Given the acknowledged difficulties of translating search strategies between interfaces, if additional searching is required for updating purposes, perhaps requests could be made to the original EUnetHTA authoring agency, to rerun the searches on behalf of the adapting agency. EUnetHTA may also wish to ensure that dedicated peer review of search strategies by information scientists takes place as well as the more general review of the report, to ensure the integrity of this critical first step of the review process. Another option to assist with the adaptation process would be for more detail to be provided of decision making around the construction of search strategies, within the EUnetHTA reports.
The additional work in exploring the device indications in the mitral valve repair was not productive in that we simply replicated the EUnetHTA statement in the absence of any new information. Similarly, the additional effort we expended in seeking publically available information to avoid citing manufacturer's submissions to EUnetHTA was also, on reflection, not essential if we assume that readers are likely to be most interested in the primary evidence sources around clinical effectiveness, cost-effectiveness, and safety of the technology rather than descriptive background information.
Reflecting on this experience, we concluded that, in future, we would not routinely undertake to supplement and update the EUnetHTA review, other than adding in the cost-effectiveness analyses. One circumstance, however, where this may be different would be if we were aware of any key studies due for publication immediately following the completion of the EUnetHTA report. In this case, we would check whether this study had been published, and also undertake a systematic search for any other studies published in the same period. Having a list of ongoing studies and their anticipated publication dates within EUnetHTA reports is very helpful in this respect.
Our underlying message to other agencies would be to work with the source review as soon as possible following the decision that it is suitable for adaptation, keep additional information gathering to a minimum, and, importantly, commit to trusting both the accuracy and the currency of the source review based upon the methods and processes in use by the source network. Both our Evidence Notes were started as the work on the EUnetHTA project was drawing to a close and they were published at 3 months (mechanical thrombectomy) and 5 months (mitral valve repair), respectively, after the publication of the EUnetHTA reports. It is difficult to specify an exact time after which an adaptation would not be feasible as this depends very much on the extent to which a technology is developing, spreading and being investigated. However, we would suggest that adaptations should ideally be undertaken within 6 months of the publication of the EUnetHTA work.
Contextual Information
The process we follow in Scotland of conducting a peer review process for all our Evidence Notes and providing peer reviewed Advice Statements to accompany these, gave us an opportunity to incorporate Scottish contextual information with the evidence. For organizations that do not include peer review, or have a separate process in place for applying local context in formulating advice, the adaptation of EUnetHTA products to meet local needs may be more challenging. These aspects of our processes also enabled us to retain local engagement with the work. The role of engaging with local stakeholders must be borne in mind as any streamlined adaptation process must not have the effect of disenfranchising those decision makers from the advice being offered. Our stakeholders did not express any negative views on the use of an adaptation process to provide advice for NHS Scotland.
There is potential for staff conducting the adaptations to feel removed from the evidence which they have not themselves identified and synthesized. This was alleviated in our case by acting as peer reviewers on the original EUnetHTA projects, and such involvement is something that other agencies may wish to consider. We also had ready access to discussion with the review authors for clarification of information.
For both the mitral valve repair and the mechanical thrombectomy Evidence Notes, the development of associated Advice Statements presented no issues specific to using an adapted product. This may partly reflect that, for both technologies, the evidence led to relatively clear conclusions. For future topics where the evidence may be more complex, greater effort in producing locally relevant advice may be required. Also, it cannot be forgotten that timeliness for decision making is key, and while these adaptations were available in line with Scottish decision-making timescales, European wide timelines may not always suit a local context. In our experience, particularly for nonmedicine technologies reviews, which are less likely to have to coincide with regulatory deadlines than medicine reviews, there can usually be scope for some movement on local decision-making timelines, However, this is one of the issues that EUnetHTA will be considering in its Joint Action 3 project.
Given the foregoing, we think it would be useful for HTA agencies to develop a defined process for undertaking such adaptations. For example, this may set out a procedure for when and if additional literature searches and information gathering should be conducted, and whether there should be a local peer review process that focuses only on specific sections of the adapted report that are likely to be more contextually different. When and how local contextual information should be added, and stakeholder engagement achieved, should also be covered.
With the increasing time pressures and specialized skills required for undertaking rapid reviews, and a growing agenda to ensure adequate assessment and appraisal of nonmedicine technologies as well as medicines, the adaptation of collaboratively prepared HTAs would seem to offer a compelling option for agencies to adopt continue to fulfil the needs of decision makers. We have shown that such adaptations can work and can be facilitated by developing and specifying agreed local processes to achieve the adapted product. The publication of work plans in advance by EUnetHTA would also aid this process.
CONFLICTS OF INTEREST
The authors declare that there are no conflicts of interest.
