The identification, filtration, prioritization, and subsequent assessment of significant new and emerging health technologies, known as early awareness and alert (EAA) activities, contributes to improved decision making processes, providing timely and useful information on relevant technologies for their adoption (or not) in healthcare systems throughout the developed world (Reference Simpson, Packer and Carlsson15). The purpose of these activities is to assist in the control and streamlined adoption and diffusion of new health technologies within health systems (Reference Douw5). However, new and emerging health technologies are not a single entity and encompass a wide range of technologies from pharmaceuticals, devices, diagnostic tests, and surgical procedures, to programs and settings (Reference Goodman8). In addition, the healthcare systems that adopt these technologies and the decision making processes for each type of technology are complex and varied. These make EAA activities multifaceted tasks that require robust and transparent methods.
The International Information Network on New and Emerging Health Technologies (EuroScan International Network) was formally established in 1999 following discussions and workshops to establish the feasibility, benefits and need for a collaboration that focused on early awareness and alert activities (Reference Carlsson and Jørgensen1;Reference Carlsson and Jørgensen2). In September 2006, EuroScan held a workshop to consider its experiences and achievements; and to discuss EuroScan's future roles, goals and collaborations (Reference Simpson, Packer and Carlsson15). Following the workshop EuroScan members agreed that they would develop a methods toolkit to summarize the various approaches used in EAA systems for agencies interested in establishing, or improving an existing, EAA system. Members also agreed to update both the work performed by Douw et al. in 2003 (Reference Douw, Vondeling, Eskidsen and Simpson4) on Internet-based sources used to identify new and emerging health technologies, and the comparative analysis on the status of and methods used by member agency systems (Reference Simpson and Packer17).
OBJECTIVE
We aim to summarize the methods and results of research activities that specifically relate to methods used within EAA systems. Our objectives are to describe the methods used to develop the EuroScan methods toolkit and present the key results, and to describe the similarities and differences in methods used by EuroScan member agencies, including the use of Internet based sources for the identification of new and emerging health technologies.
METHODS
EuroScan Methods Toolkit
The approach used to develop the toolkit was based on the Delphi technique. This technique has been described as a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem (Reference Linstone and Murray11).
All members of EuroScan, the “expert panel”, contributed to the content of the toolkit. A first draft was proposed by one of the EuroScan member agencies followed by 4 rounds at which 3–4 member agencies were given the opportunity to add to, or comment on previous versions. After each round the feedback was collated and the content of the toolkit edited by the EuroScan Network Coordinator (SS). Care was taken to include all feedback and communicate all suggestions whilst maintaining a document that was suitable to pass on to the next round of recipients. After all agencies had been consulted in the individual rounds, the document was edited and a summary checklist developed in the form of questions to be considered by those seeking to establish or review an EAA system. This was then sent to the whole panel for further comment and revised following feedback. The final document was reviewed by all EuroScan members in April 2009, before publication in June 2009.
Figure 1. Stages involved in early awareness and alert systems.
Comparative Analysis of EuroScan Members EAA Systems
A comparison of EuroScan member agency structure, processes and outputs was carried out in February 2008, and updated in early 2009. Member agencies completed a questionnaire comprising forty-two questions about their EAA system, including questions on structure and funding, aims and coverage, customers, partnerships and collaborations, methods, output, dissemination, related activities, and future developments. The questionnaire, sent by email, included a combination of open-ended and closed questions (see Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012034, for more details). Reminders were sent to non-responding agencies.
A supplementary questionnaire listing possible sources available on the Internet for the identification of different types of new and emerging health technologies was developed and sent to all EuroScan members in June 2008. The list was derived using previous research findings in this area (Reference Douw5) and the researchers’ experience. The questionnaire was presented as a list of sources grouped by type including Web sites of HTA agencies, EAA systems, health organizations, related organizations, marketing authorization agencies, news sites, and journals. Respondents were asked to indicate if they used the source to identify technologies or specific technology types, for example, devices, diagnostics, drugs, procedures, programs, and settings; and to say how relevant the source was assigning a value from 1 to 9 (from most to least relevant). A score of 0 was assigned if the source was not used. Respondents were also given the opportunity to add new sources. In total, three reminders were sent to agencies not replying to the initial request(s) to complete the questionnaire. Once the completed questionnaires were received, the median score for each source was calculated and the sources were ordered according to the number of votes and the median score they obtained.
RESULTS
EuroScan Methods Toolkit
The toolkit resulting from the consultative process (Reference Simpson, Hiller and Gutierrez-Ibarluzea16) incorporated guidance on all of the stages involved in an EAA system (Figure 1). These stages were not all carried out by all member agencies and when they were carried out agencies often document different approaches.
Twenty-one key questions for those seeking to establish or review an EAA system were developed (Table 1 and supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc2012034). The final toolkit is available on the EuroScan International Network Web site (http://www.euroscan.org.uk) and two workshops based on the structure and content of the toolkit have been delivered to international audiences at Health Technology Assessment international (HTAi) conferences in Dublin 2010 and Rio de Janeiro 2011.
Table 1. Checklist to Establish an EAAS
Comparative Analysis
All EuroScan member agencies responded to the initial questionnaire (n = 20). Of these EAA systems, 90 percent considered devices and diagnostics, 80 percent procedures, 70 percent pharmaceuticals, 60 percent programs, and 50 percent health care settings. All but one EAAS considered all disease areas (95 percent). The system that had restricted coverage specifically identified advances in the oncology field only.
Identification
All agencies had an identification stage in the EAA system, but the amount of scanning of sources carried out by agencies to identify new and emerging technologies varied along with the type of sources. Eleven (55 percent) agencies used primary sources available on the Internet. Two (10 percent) agencies obtained information by liaising with developers/manufacturers of the technologies. Fifteen agencies (75 percent) either relied on or supplemented the primary scanning sources with a more reactive approach where technologies are suggested to them via a Web-based form, input from expert groups or an open call is made for suggestions.
Completed supplementary questionnaires on Internet-based identification sources were received from 13 of the 15 EuroScan member agencies. Identification sources that obtained low median scores (a low score indicated a relevant source), for each technology type (devices, diagnostics, drugs, procedures, programs, settings) are shown in Table 2 and Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2012034. Of the original list of sixty-two identification sources provided in the questionnaire, twenty-two were scored as highly relevant or relevant sources and included in the tables. Some sources are used for the identification of all types of technologies, for example, medical journals (BMJ, Lancet or JAMA); scientific societies like American Cancer Society or Internet alert facilities such as Medscape, whereas others are used to identify specific technology types, for example, European Medicines Agency (EMA) for the identification of new drugs. The EuroScan database of new and emerging technologies was regarded as highly relevant for the identification of all types of technologies and was used as a source of identification by most respondents. This was expected as the technologies on the database are all emerging or new technologies that have been prioritized by other EAA systems.
Table 2. Sources for the Identification of New and Emerging Technologies (1 = Most Relevant, 9 = Least Relevant)
Table 3. Content of EAA Reports Relating to Impact Prediction and Assessment
Filtration
All agencies filtered the technologies they identify, to reduce numbers of health technologies to investigate and assess to a manageable size; with 18 saying they had a formal filtration process. Filtration was mainly carried out in-house (60 percent), by an expert scientific board (25 percent), by the policy-making or other customer (15 percent) and/or by individual experts (10 percent). Some agencies have explicit criteria based on methodological documents (Reference Carlsson3;Reference Noorani, Husereau, Boudreau and Skidmore13;Reference Rico and Asua14).
Prioritization
Most agencies used prioritization criteria (90 percent). Commonly used criteria were: patient numbers/burden of disease, potential clinical benefit, possible economic impact and anticipated speed of adoption (both appropriate and inappropriate). Other criteria considered to a lesser extent were level of available evidence, existence of alternative technologies, safety profile of the technology; and social, ethical and legal aspects. Agencies also stated national, regional and local priorities as having an influence on the prioritization of technologies for further consideration.
Assessment and Dissemination
Agencies produced a range of outputs with different levels of impact prediction and assessment (Table 3). The extent to which technologies were assessed also varies with the timeframe covered, with those agencies considering technologies further away from launch having less evidence to include in their reports. Respondents generated a variety of report types - short (1–4 pages), long (4–10 pages), comprehensive (>10 pages) and newsletters. These were disseminated by e-mail (70 percent), Web sites (50 percent), paper (30 percent), and medical journals (5 percent). Two agencies (10 percent) produced reports for internal use only which are not disseminated externally.
Collaboration
In carrying out their EAA activities agencies worked with clinical or scientific individuals (90 percent), existing expert groups (70 percent) or specially convened expert groups (50 percent). Forty-five percent of respondents contacted commercial companies, developers or sponsors in their work, mainly for identification of technologies. Some agencies invited commercial companies and clinicians to comment on the final assessments as part of the review process.
DISCUSSION
Both the development of a methods toolkit and the comparative analysis of EuroScan member agencies indicate that methods used by existing EAA systems are not homogeneous. Agencies all use methods that involve some form of identification, filtration and/or prioritization, and assessment but these differ depending on customer requirements and desired outcomes, resources available and the time horizon being considered (Reference Ibargoyen-Roteta, Gutierrez-Ibarluzea, Benguria-Arrate, Galnares-Cordero and Asua9).
The EuroScan methods toolkit has brought all the approaches to EAA activity together in one document for the first time. The collaborative approach to the development of the toolkit resulted in an end product covering all aspects of EAA processes as well as presenting key questions and options for consideration by anyone developing or modifying an EAA system. The Agency for Healthcare Research and Quality (AHRQ) Healthcare Horizon Scanning System has recently published an Horizon Scanning Protocol and Operation Manual that outlines the basic protocol and decision processes that are being followed. It covers scanning to identify leads for new interventions, selection of topics for in-depth information searches, and identification of interventions that could have the greatest potential impact in each priority area (6). The main merit of the EuroScan toolkit as a starting point in comparison with this and other individual protocols of EuroScan member agencies is that it includes different approaches that can be adopted to lead to a context tailored protocol.
The usage of a Delphi technique to develop a collaborative document from a heterogeneous group is not unique to EuroScan, consensus methods have been used in other methodological guidelines or toolkits in HTA (Reference Márquez Calderón, Castilla Alcalá, Briones Pérez de la Blanca and Carriazo Pérez de Guzmán12). The modified Delphi approach proved to be successful and enabled a toolkit to be constructed that represented the views, methods and contexts of all member agencies. This is an advantage over commentaries on methods provided by a single agency. The toolkit is not presented as a directive instruction manual but allows the reader to consider several methods and adopt the approach that suits their needs.
The updated comparative analysis of methods confirmed that agencies do follow the principle EAA steps outlined in the toolkit and shown in Figure 1 but that their approach to the steps may be different. It could be said that the similarities illustrate a shared understanding of EAA activities that has developed though the collaborative efforts of EuroScan. However, the variations between systems are inevitable in an international collaboration where health care systems and services operate differently and the end products feed a diverse group of customers with varying needs.
The supplementary questionnaire on Internet-based identification sources highlights a resource intense process where agencies differ in their approach. Although systems mainly use the Internet as an information source to identify new technologies, the evidence suggests that their use varies and is not homogeneous, despite access to sources, on the whole, being available to all. In fact, previously published research points to the existence of a large number of potentially useful sites and that their use may be based on local preferences by information specialists. This may be critical to the outcome of the identification process (Reference Douw5).
It is also clear that identification sources are not static, new sources are introduced all the time each varying in their coverage, relevance, quality, cost, frequency and ease of use. Agencies use what provides them with the information most relevant to their needs, resources and the context in which they and their decision makers operate. Other research has looked at how to assess the value of identification sources for horizon scanning (Reference Smith, Cook and Packer18), using generic databases such as Medline for identification (Reference Varela-Lema, Puñal-Ruiboo and Casal-Accion19), and sources for the identification of innovative public health interventions (Reference Fung, Simpson and Packer7). The value of these efforts and their results should be considered by any agency undertaking EAA activities. EAA systems’ identification, filtration and prioritization methods are also potentially transferable to other areas in health service decision making. This may not only be related to new and emerging technologies but could be of value when exploring or identifying obsolete or low added value health technologies (Reference Ibargoyen-Roteta, Gutierrez-Ibarluzea, Asua, Benguria-Arrate and Galnares-Cordero10).
Although heterogeneity amongst a group of individuals and agencies working together can cause difficulties, one of the strengths of collaboration is that the different methods, experiences and skills of individual members can be shared and enhance the knowledge and practices of other members. Through this research EuroScan members have embraced these differences in approach to EAA work and have developed a strong shared understanding of EAA systems and activities.
The EuroScan International network increased its membership from 12 in 2006 to 20 in 2010, which has, not only improved its capability to face more complex activities and strategies, but has enriched the methods used and the experiences shared. The weakness of the research carried out is that it does not consider other agencies and systems who are not members of EuroScan and who may operate in a very different way. Future work in EuroScan could concentrate on developing a consistent approach to the identification of new and emerging health technologies for each technology type. This could be supplemented by relevant local sources and a more proactive collaboration with international and local producers and local users of health technologies.
CONCLUSION
Methods used in EAA systems continue to develop with the emergence of innovative technologies such as genomics and regenerative medicines, new pressures on health systems, and changes in the decision making processes that EAA systems feed. Differences between agencies in approaches and methods with the sharing of ideas and experiences, will continue to enrich these developments and ensure that decision making processes in healthcare systems continue to be supported.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2
Supplementary Table 3
CONTACT INFORMATION
Iñaki Gutierrez-Ibarluzea, BSc, MSc, MD in Bioethics, PhD, Osteba Basque Office for Health Technology Assessment, Department of Health and Consumer Affairs, Basque Government, Vitoria-Gasteiz, Araba, Basque Country, Spain
Sue Simpson, BSc (Hons), PhD, MPH, Associate Director, NIHR Horizon Scanning Centre, School of Health and Population Sciences, University of Birmingham, Birmingham, United Kingdom
Gaizka Benguria-Arrate, BSc, MSc, Osteba, Basque Office for Health Technology Assessment, Department of Health and Consumer Affairs, Basque Government, Vitoria-Gasteiz, Basque Country, Spain
CONFLICTS OF INTEREST
The authors report they have no potential conflicts of interest.
