New health technologies are a major pressure on health systems, with many concerns about their cost, the timing and appropriateness of use in relation to the evidence base, and the number of technologies in development. New health technologies need prioritizing for additional research, assessment or evaluation; the development of guidance to health services, professionals, and patients; and also for financial and service planning.
Many health services have early warning or horizon scanning systems in place to provide advanced notice of emerging health technologies to policy makers enabling them to act in advance of the technology becoming available in the health service. Although current early warning systems have variable health service contexts, customers, and integration into policy making, the core steps of identification, information gathering, filtration and prioritization, and early assessment are similar (3).
Although early warning systems have been in place for several years, there are as yet no criteria or common understanding of what makes an early warning system effective. There are two published studies of the accuracy of past prioritization, but no published research into overall effectiveness (2;4). In the broader context, there is one published review of evaluations of health technology assessment agencies, and a model for evaluation of agencies and programs for health technology assessment that identifies opportunities for future improvement (1;5). Hailey's model sets out categories of determinants of effective health technology assessment programs that include the quality of the product, effective governance and mandate of the program, the staff and structure of the program, collaborative and contractual relationships, and influence on the primary targets of the assessments. Within each of the categories, Hailey provided examples of issues that may need to be considered; so for instance, in the area of staff and structure of the program, any evaluation could consider what technical competence is required to produce an assessment, the experience and competence of the assessment staff, and the morale and stability of the program.
In this study, we set out to describe the characteristics associated with effective early warning systems for emerging health technologies and to evaluate current early warning systems against these characteristics.
METHODS
We used representatives from the thirteen member agencies of the International Information Network on New and Changing Health Technologies (EuroScan, formally the European Information Network on New and Changing Health Technologies) in a Delphi-type process to develop a definition of an effective early warning system, to identify and rank potentially important characteristics or components of effective early warning systems, and subsequently to evaluate their own systems against these characteristics. For more information about EuroScan, its aims and members, see www.euroscan.bham.ac.uk.
Definition
Based on our experience in early warning activities and a search of the literature, we identified two potential components of an effective early warning system: providing timely advice to allow appropriate implementation and/or adoption of health technologies, and to facilitate timely budgetary planning. These possibilities were sent in our first mailing to participants for them to consider and augment. We added themes coming from respondents to the initial mailing round and developed a final definition.
Ranked Characteristics
Based on our experience in early warning activities and informed by a search of the literature, we identified nine characteristics or components of early warning systems that could relate to their effectiveness. The characteristics and components were divided into structure, process, and outcome-related groups and sent in the first mailing to participants for grading according to importance and for the addition of any characteristics or components thought to be missing. We asked respondents to imagine that they were responsible for setting up a new early warning system and to grade the characteristics using a Likert scale of importance: 1, high importance to an effective early warning system; 2, moderate importance to an effective early warning system; 3, low importance to an effective early warning system; and 4, no importance to an effective early warning system. Additional characteristics and components suggested by respondents were returned to participants in a second mailing along with the group median scores for each of the original characteristics. Participants were asked to grade the additional characteristics and could re-grade the original characteristics if they wished.
We calculated the final median scores and ranked each characteristic. Characteristics with a median score of 1.0 we labeled as “primary” characteristics, and those with a median score of more than 1.0, we labeled as “secondary” characteristics. We calculated the proportion of respondents who scored each characteristic as being of high importance, and compared it with a 95 percent confidence interval around the proportion of all characteristics that were scored as of high importance to identify characteristics of particular significance or irrelevance.
Evaluation of Early Warning Systems
We asked participants to evaluate their own early warning system against the primary and secondary characteristics in a final mailing round. We asked participants whether each characteristic was: 1, definitely or completely present or fulfilled in their early warning system; 2, largely present of fulfilled; 3, present or fulfilled to some degree; 4, definitely not present or fulfilled; or 5, not applicable to their system. We looked for correlations between variables using a two-tailed Kendall's tau correlation coefficient calculated using SPSS 12.0.1 for Windows. Due to multiple correlations, we used a .01 significance level.
Nonresponders to the first two mailings were chased once by email. Nonresponders to the evaluation of early warning systems who had responded to earlier mailings were followed-up by telephone.
RESULTS
Eleven responses of a possible thirteen were received for the first mailing, ten for the second mailing, and twelve for the evaluation of early warning systems against the ranked characteristics.
Definition
The final definition of an effective early warning system incorporating all themes is in Table 1.
Ranked Characteristics
Table 2 shows the final ranked characteristics and components (five primary and eleven secondary), with their median and mean importance scores. No characteristics moved between the primary and secondary categories after re-grading by participants. Four of the five primary characteristics are related to the structure of the system, the fifth is an outcome-related measure. One of the secondary characteristics is a structural measure, eight are process measures, and two are outcome-related measures.
Figure 1 shows the proportion of respondents scoring each characteristic as of high importance, with the average proportion and its higher and lower 95 percent confidence limits. The proportions for all five primary characteristics are above the upper 95 percent confidence limit. Four secondary characteristics lie below the lower 95 percent confidence limit.
Proportion of respondents scoring each characteristic as of high importance with the group average proportion and its 95 percent confidence interval.
Evaluation of Current Early Warning Systems
All primary characteristics were reported to be present or fulfilled to some extent in all systems with the exceptions of sufficient funding and staffing, and having defined customers for early warning system outputs, which were each rated as not present or fulfilled in two early warning systems (Figure 2). All early warning systems were independent of industrial and commercial influences. Nine systems considered that their output was largely or completely relevant to the needs of their customers and that their system had largely or completely defined customers or was integral to the national policy-making process.
Presence or fulfillment of primary characteristics by early warning systems (EWSs).
There was more variation in the presence or fulfillment of the secondary characteristics and components (Figure 3). Characteristics completely or largely present or fulfilled in two thirds or more of the systems included having explicit, agreed methods and criteria for prioritization; having explicit, agreed methods and criteria for identification; filtering and prioritization of technologies undertaken in-house; and having a fixed panel of experts to consult. Characteristics reported to be not present or only present to some degree by nine or more systems included the ability to evaluate the usefulness of the early warning system to its audience, collaboration with industry to obtain information, close contact with industry to identify new technologies, and sufficient budget for decisions on emerging or new health technologies to be implemented appropriately. Two of the four characteristics that fell outside the lower confidence limit of importance: in-house filtration and prioritization, and having a fixed panel of experts, were completely or largely present in ten and eight systems, respectively, a greater proportion than more highly rated secondary characteristics.
Presence of fulfillment of secondary characteristics by early warning systems (EWSs). N/A, not applicable.
We found three significant positive relationships at the .01 level between (i) the production of information relevant to customers and having defined customers or a system that is integral to the national policy-making process (Kendall's tau .749; p = .006); (ii) having a clearly defined route or pathway for outputs to reach local and national decision makers and having defined customers or a system that is integral to the national policy-making process (Kendall's tau .792; p = .003); and (iii) having explicit, agreed methods and criteria for prioritization and clinical or technology experts involved in the filtration, selection, or prioritization process (Kendall's tau .686; p = .001).
DISCUSSION
Our study provides a definition of an effective early warning system and adds to the limited literature on methods for the early identification and early assessment of emerging health technologies. The definition appears to have face validity in that it incorporates key elements about the selection of significant technologies, and the provision of relevant and timely information for use by decision makers and researchers. We have also developed a ranked set of characteristics and components of early warning systems that are likely to be associated with an effective system, and piloted these components on twelve publicly funded early warning systems. In the self-evaluation, the majority of the primary characteristics were present or fulfilled to some extent in all the early warning systems. A greater degree of variation was seen in the secondary characteristics.
Four of the five primary characteristics are obvious ones for success of any organization. All effective information systems must produce relevant information for a clearly defined target audience, and have clearly defined routes for their outputs to reach their audience. An even more effective situation is where the outputs of the information system are integral to later decision-making processes. Independence from potential lobby groups and sufficient resources to undertake the work to the required standard are also likely to be important to health-related policy makers who have to make decisions on behalf of the whole population.
Strengths and Limitations
The use of the EuroScan member agencies as our study set was a pragmatic decision in that, although there are experts in the field of horizon scanning and early warning outside the EuroScan collaboration, the collaboration is likely to be a nearly complete set of publicly funded active agencies. It also made for a readily available and accessible group of willing participants.
Despite the likely completeness of the network, this reliance on the expertise and experience of EuroScan members could have led to subjective rather than objective responses in both the development of the ranked characteristics of effective early warning systems and in the self-evaluation against these characteristics. The development of the characteristics and the self-evaluation would also have been influenced by the context within which each early individual system operates, with differences in outputs and policy-making customers of each system leading to differences in the methods used. The definition will also reflect the context and customers of the systems. The EuroScan agencies are publicly funded and characterized independence as important; industry funded agencies might have thought otherwise. Systems that identify topics for further research proposed the addition of the research requirement component.
We tried to remove some subjectivity in the development of the ranked list of characteristics by asking respondents to imagine they were setting up a new system, although we do not know to what degree this was achieved. There may have been a tendency to either portray a system or country in the best light possible or alternatively to portray a system as needing more resources or re-structuring.
Comparison with Other Studies
Although we did not set out to constrain our respondents with predetermined categories, some of our ranked characteristics and components map into Hailey's categories for the determinants of effective health technology assessment programs. Characteristics such as the system's independence from industrial or commercial influences fits within Hailey's governance and mandate category, relevance of the output to the customer fits within the quality of the product category, and sufficient funding and staff within the staff and structure category. Our characteristics of having defined customers or being integral in the decision-making system, and having a clearly defined route for the outputs to reach decision makers relate to Hailey's category about having influence on primary targets.
The relative lack of outcome-related characteristics and components, although of some concern, is consistent with ideas around evaluation of systems and organization. When organizations are relatively new, evaluation is likely to be inputs and structures and processes. As systems mature, the evaluation focus can shift toward program outputs and impact, with a view to measuring a programs' merit or worth (5).
CONCLUSIONS AND POLICY IMPLICATIONS
We have developed a definition for an effective early warning system and a shared understanding of the important characteristics and components of such systems. This work is of relevance and can provide guidance to those setting up new early warning systems as well as for those managing and reviewing current systems. The self-evaluation of current early warning systems against these characteristics and components, although subject to some bias, provides an insight into the possible effectiveness of current systems and identifies opportunities for possible improvement.
CONTACT INFORMATION
Kieran Murphy, BMedSc, MPH, Research Associate, Claire Packer, BM, BS (c.packer@bham.ac.uk), Senior Clinical Lecturer in Public Health, Andrew Stevens, MB, BS (a.j.stevens@bham.ac.uk), Professor of Public Health, Sue Simpson, BSc, MPH, PhD (s.l.simpson.20@bham.ac.uk), Research Fellow, National Horizon Scanning Centre, Department of Public Health and Epidemiology, The University of Birmingham, Edgbaston, Birmingham, B15 2TT United Kingdom
We acknowledge and thank the EuroScan collaborators who provided data: Janet Hiller, Brendon Kearney (Australia); Jill Sanders, Leigh-Ann Topfer (Canada); Birgitte Bonnevie, Mads Frellsen (Denmark); Claire Packer, Sue Simpson, Andrew Stevens (England); Anne-Florence Fay (France); Gabriël ten Velden (Netherlands); Inger Norderhaug (Norway); Iñaki Gutiérrez, Setefilla Luengo, Román Villegas (Spain); Per Carlsson, Helen Törnqvist (Sweden); Christoph Künzli (Switzerland). At the time of the study, Kieran Murphy and Claire Packer were funded by the Research and Development Department of the Department of Health for England; Andrew Stevens by the National Health Service for England; and Sue Simpson by EuroScan and the Research and Development Department of the Department of Health for England.
