The National Institute for Health and Clinical Excellence (NICE) is established within the National Health Service (NHS) as an organization that makes evidence-based decisions regarding the effective use of NHS resources (Reference Devlin, Parkin and Gold2). It aims to produce guidance based on the best available evidence and using processes that are transparent, collaborative and involve its stakeholders. In addition, the methods NICE adopts are subject to regular and close scrutiny, and have been the subject of some criticism (Reference Drummond and Sorenson3;Reference Littlejohns, Garner, Doyle, Macbeth, Barnett and Longson5). For example, in a review of NICE by a UK parliamentary Health Select Committee in 2007, it was noted that, although NICE's evaluation procedures had been shown to be generally robust, problems were perceived in the methods it uses and the need for further methodological research was communicated (4). Areas for improvement included topic selection, the evidence base available to NICE, the timeliness of the publication of guidance and the measurement of the costs and benefits of health interventions. Similarly, in his review of UK health research funding, Sir David Cooksey (Reference Cooksey1) also noted the need for further research into health technology assessment (HTA) methodology to support the work of NICE.
NICE has four main guidance producing programs: clinical guidelines; technology appraisals; public health; and interventional procedures. One important difference between these programs is that the interventional procedures program focuses on the safety and efficacy of interventions, but does not include a consideration of cost-effectiveness, whereas this information is routinely considered in the other three programs. Methods guides for each program describe the analytical techniques and processes used to produce NICE guidance. NICE has stated a commitment to using the most up-to-date methods for its evaluations (8) and the methods guides are reviewed regularly (Reference Longworth and Longson7;Reference Sculpher10). As with similar healthcare evaluation agencies, the NICE methods review process may highlight specific methodological areas that require further research. However, a comprehensive review of priorities for methodological research to help facilitate NICE decision making has not previously been conducted. The aim of this review was to establish an initial list of priority methods research topics to support NICE decision making. This review was conducted as part of a broader project which also included an evaluation of the processes by which such priorities are identified and prioritized by NICE; the results of this aspect of the project are reported elsewhere (Reference Longworth, Bojke, Tosh and Sculpher6).
METHODS
Overview
A variety of methods were adopted to obtain the views of a range of NICE stakeholders with an interest in evaluation and NICE. A focused literature review aimed to obtain information on NICE's research needs as reported by evaluation experts and from government-related reviews relating to NICE's work. Interviews were used to obtain detailed information from key members of NICE and organizations closely allied to it. An email survey was used to capture the views of a broader group from NICE, its stakeholders and HTA and health evaluation organizations within and outside of the UK. Finally, a workshop was convened and a Web-based exercise conducted to get feedback on a long list of topics collated at each stage of the project.
Topic suggestions were limited to research into methods to support NICE decision making. A reasonably broad view of what constitutes methodological research was adopted; however, purely empirical research relating to specific health conditions was excluded. Suggestions for topics came from each element of the project and duplicate suggestions were deleted or merged where this was possible.
Literature Review
A focused literature review was conducted to identify potential methods research topics relating to NICE's decision-making activity. The review included key policy documents identified by the project team and its advisory group. The review of the literature included recent key policy documents relating to the work of NICE and a focused review of NHSEED (which searches MEDLINE and EMBASE) and the Cochrane Methodology register. Search terms were restricted to methodology, method, and methods and papers published between 2004 and 2008. This was supplemented by a review of the literature conducted by NICE for its technical staff (the “What's New” bulletin), reviews of abstracts from NICE conferences and articles identified by the extended project team. In addition, the NICE Methods Guides and supporting documentation from the 2008 review of the NICE Technology Appraisals were reviewed (comparable supporting documentation was not available for the other NICE programs). Full details of the search strategies are reported elsewhere (Reference Sculpher6).
For inclusion, papers had to address methodological issues explicitly related to NICE decision making and/or the processes used at NICE to develop or promote methodological research. All titles and abstracts were reviewed by two reviewers and any disagreement resolved through discussion.
Interviews
A series of interviews was conducted with representatives from NICE (including the Directors of each of the guidance programs, the R&D program and the Patient and Public Involvement Program), its advisory bodies and representatives from organizations that work closely with NICE and/or provide evidence to inform its decision making (for example, the Association of British Pharmaceutical Industries and the National Institute for Health Research Health Technology Assessment program). Interviewees were identified by the project team in collaboration with advisors from NICE. Interviews were semi-structured and conducted by telephone or in-person. Interviewees were asked to suggest up to five topics for methods research that they considered to be most important to assist the NICE decision-making process. They were also asked why the suggested topics were important and how they could contribute to NICE's work.
Email Survey
The interview schedule was translated into a survey which was sent by e-mail to a wider group of thirty-three people that work with NICE. Recipients of the email survey were agreed with NICE and included Associate Directors and senior technical staff working at NICE, the academic groups who conduct reviews of the evidence for NICE, representatives of key evidence/research collaborations in the United Kingdom (such as the Cochrane Collaboration), and representatives of a selection of similar organizations outside of the UK (such as the Agency for Healthcare Research and Quality in the United States).
Workshop
A workshop was convened with the objective of clarifying and refining the list of potential methodology research topics identified by the focused literature review, interviews and email survey. Participants included members of NICE staff and its advisory committees, representatives from the academic groups conducting evidence reviews for NICE, industry, and methodologists. In preparation for the workshop, topics were grouped into key areas of methodology by one of the authors (L.L.) and reviewed by the rest of the project team. Consensus about any differences was reached through discussion. Topics that related to more than one area were categorized into to the one which was considered most relevant.
Following a series of presentations on the general topic areas, participants discussed some of the topics in small groups. Participants were specifically asked to consider which of the potential research topics were most important and to discuss the topics in relation to the following five key questions. (i) Do they address challenges faced by NICE? (ii) How can they add value to NICE's role? (iii) Can questions be further defined? (iv) Are there additional topics? (v) What should be the initial prioritization?
Web-Based Feedback Exercise
The categorization of topics into methodological areas was revised following feedback during the workshop. The final long list of topics was made available by means of the UK Medical Research Council (MRC) Web site in the form of a feedback questionnaire. A link to the Web site was sent to NICE staff, members of NICE advisory bodies, and those invited to participate in earlier stages of the project, including the workshop. The feedback questionnaire was also publicly advertised on the NICE and MRC Web sites. Respondents were asked to rate each topic on a scale of very important, important, fairly important and not important. They were also given the opportunity to nominate any other topics. Respondents were asked to rank their top three priorities and to provide information on how each could help improve decision making at NICE. Equal weight was given to the response of each respondent.
Further Analysis
Each stage of the project involved slightly different groups of individuals, some of whom work directly with NICE to varying degrees: the interviews focused on NICE and those involved closely with it; the email survey included larger group of these people, researchers providing evidence to inform NICE decisions and representatives from other similar agencies; in addition to these groups, the workshop and Web-feedback exercise included broader groups of people from industry and academia. Therefore, the responses at each stage of this review are analyzed separately. The following definitions were used to identify high priority topics at each stage of the project: (i) Topics independently suggested by two or more people during the interview stage (the group closest aligned to NICE); (ii) Topics raised more than twice as most important during the group discussions at the workshop (slightly broader group including methodologists with links to NICE); (iii) Topics that were rated as important or very important by 75 percent or more of respondents (the broadest group, which included people who may have no direct professional links to NICE).
The relevance of each topic in relation to each of the NICE guidance producing programs and in relation to key activities in production of guidance was considered. The topics were categorized by one of the authors (L.L.) according to their relevance to one or more of the four NICE guidance programs. In addition the topics were again categorized according to one of six key activities within the guidance production process they most closely related: (i) scoping the question to be addressed by the guidance, (ii) evidence synthesis, (iii) primary data collection and/or analysis, (iv) economic analysis, (v) engagement with stakeholders and; (vi) decision making. The categorization was checked by other members of the project and any disagreement resolved through discussion.
RESULTS
Literature Review
A total 1,150 journal articles were screened. After initial screening, nineteen journal articles were reviewed in full alongside twenty-two policy documents and NICE process documents. Of these, eleven specific suggestions for methods research topics explicitly addressing the work of NICE were identified.
Response Rates
Eighteen interviews were conducted. Of thirty-three invited email-survey respondents, thirteen (39 percent) completed surveys were received. Although the response rate is low, there was a reasonable response rate from NICE employees (60 percent), UK HTA and health evaluation methodologists, and representatives from academic groups conducting evidence reviews for NICE (75 percent). Two representatives from other evidence/research collaborations responded that they did not believe that they were sufficiently informed regarding NICE to complete the survey. Seventy-six responses to the Web feedback exercise were received. Details of responses to the interviews and email survey are provided in Table 1. Most respondents to the Web feedback exercise were based in universities (66 percent); others included NICE (13 percent), pharmaceutical industry (9 percent), the NHS (7 percent), and others (5 percent). Their professional backgrounds included health services researchers (32 percent), health economists (26 percent), clinicians (13 percent), public health specialists (9 percent), and others (20 percent). Further details on the respondents are reported elsewhere (Reference Longworth, Bojke, Tosh and Sculpher6).
Table 1. Number of Invitations to Participate in Interviews and Email Survey and Responses by Type of Organization
Research Topic Suggestions
Over 100 topics for potential methodological research were identified through the interviews, email survey and focused review of the literature. There were similarities and overlap between some of the suggestions. To accommodate this and to facilitate discussion at the workshop, the suggested topics were grouped into themes under five general areas of methodology. The general areas of methodology and themes under which each of the topics were grouped are shown in Box 1 and the full list of topics is shown in Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2011011.
Box 1: Areas and Themes under which Methodological Research Questions Were Grouped
Relevance of Topics to NICE Activities and Programs
Approximately 45 percent of topics were relevant to all four of NICE's programs. A further 38 percent of topics suggested research aimed at the three guidance producing programs that routinely consider cost-effectiveness as part of their processes (clinical guidelines, public health, and technology appraisals). The remainder of topics were specific to one or more of the NICE programs.
The largest proportions of topics most closely related to economic analysis (41 percent) and evidence synthesis (32 percent). Activities relating to decision making (either the process of decision making or the considerations that should be taken into account when making decisions/recommendations) accounted for 19 percent of topic suggestions. Scoping, primary data collection/analysis, and engagement with stakeholders accounted for only 2 percent, 2 percent, and 5 percent of suggestions, respectively. Further details on the relationship between the topic suggestions and NICE programs and activities are reported elsewhere (Reference Longworth, Bojke, Tosh and Sculpher6).
Priority Topics
The topics emerging as priorities from each stage of the process are shown in Table 2. No topics were consistently prioritized at all three stages of the project; however, some were common to more than one stage. For example, methods for determining empirically the cost-effectiveness threshold were identified as a priority in the interviews and workshop. Methods for conducting efficient systematic reviews, extrapolating costs and benefits, and capturing wider benefits outside the QALY were all identified by the workshop participants and responders to the Web survey.
Table 2. Topics Emerging as Priorities at Different Stages
Table 3 shows the topics most frequently rated as important or very important in the Web feedback to the type of organization to which the respondent belongs (NICE, university, Industry, other). Although based on small numbers, it shows that those topics considered most important differ between the types of organization. Industry and “other” respondents (who were mainly from the NHS) rated a larger number of topics as important. There was consensus among the NICE respondents that research to establish when NICE should recommend a treatment in the context of research is important. However, only 63 percent of the other respondents considered this to be important. Establishing efficient methods for conducting systematic reviews was considered important by 92 percent of NICE respondents, but by only 63 percent of the non-NICE respondents. In addition, those responding to the Web-feedback exercise requested respondents to list their top three research topics. The ranking section of the feedback questionnaire was completed by 72 percent of respondents. The most common topics ranked in the top three were: methods to evaluate interventions to change behavior in the context of public health (n = 13), Methods for conducting systematic reviews of complex interventions (n = 7) and establishing efficient methods for conducting systematic reviews (n = 6).
Table 3. Most Frequently Prioritized Topics in the Web Exercise by Type of Respondent (Percentage Reporting ‘Very Important’ or ‘Important’)
aNumber of respondents that completed any of this section.
DISCUSSION
A list of potential research methodological research topics to improve the methods used by NICE to produce guidance has been collated. Across the five areas into which topics were arranged, there appears to be a broad relevance across the range of NICE's decision-making activities. Research priorities were identified by different groups of individuals, with varying degrees of links to NICE, at specific stages of the project. There were some differences in the topics identified as most important between the groups.
The project was conducted within tight time and resource constraints to inform a specific call for research funding by the NIHR MRC methodology panel. The literature review was highly focused to achieve this. Ideally the review would have extended to literature relating to processes and methods priorities for organizations similar to NICE internationally and possibly to the extensive methods literature relating to the evaluation of health services and medical technologies. However, it is unclear whether the methods research topics that might have emerged from this additional work would have directly applied to NICE's activities. In addition, the other stages of the review were used to identify any topics missed from the literature review. The Web-based survey to prioritize potential topics was limited by time—it was available for completion for a period of a month. It is likely that this time constraint, and the extensive length of the questionnaire which reflected the number of potential topics identified in earlier stages of the project, limited the number of responses for analysis. Despite these limitations, the Web comment provides some indication of how important the suggested topics are as judged by a broad group of individuals with an interest in NICE and/or evaluation methods.
Whereas the low response rate to the email survey from international HTA agencies and other research organizations may limit the generalizability of the findings to bodies other than NICE, some of the methods research areas identified are likely to be relevant to other organizations. The extent to which the identified topics may also be priorities for other national and international HTA and health evaluation bodies (such as European network for Health Technology Assessment – EUnetHTA) will depend on the similarity of methods of evaluation to NICE and of the political, legal, and cultural frameworks within which they operate.
There are numerous potential research topics that could be commissioned to help NICE decision making. In terms of setting research priorities, this study has highlighted the variation in research priorities between different groups of stakeholders with an interest in NICE decision making. Given this variation, there is a need to consider carefully whose priorities matter when funding research. The priorities of the recipients of research (e.g., the decision maker) may differ from those who use the methods (e.g., academics or industry). The NIHR-MRC Methodology Research Program has already provided funding to take nine of the identified topics forward (9). The definition of methodological research was raised throughout the process and needs to be considered as these topics are developed further. A large proportion of suggested topics are clearly methodological in that they relate to the further development and refinement of evaluative and analytical tools, which have relevance beyond a single application. Other topics may be judged less obviously methodological. For example, the estimation of NICE's cost-effectiveness threshold might be considered empirical rather than methodological, but there are likely to be several methodological questions that relate to this research. This needs to be further discussed by research commissioners to avoid ignoring those topics that do not fit into traditional categories of research.
CONCLUSIONS
We reviewed the methods research priorities to improve NICE decision making through a range of qualitative methods and a Web-based prioritization exercise. We found five key areas of methodology for which research is needed to support NICE decision making: evidence synthesis (including indirect comparisons and the use of disease registers), the use of qualitative evidence, the use of QALYs, establishing the threshold for cost-effectiveness, and how uncertainty in the evidence base should be reflected in NICE recommendations. Some research funding has already been made available to pursue some of the identified topics. Research commissioners should consider making further funding available, including for those topic areas that do not fit into traditional categories of research.
CONTACT INFORMATION
Louise Longworth, PhD (louise.longworth@brunel.ac.uk), Reader in Health Economics, Health Economics Research Group, Brunel University, Uxbridge, Middlesex UB8 3PH, UK
Mark J. Sculpher, PhD (mark.sculpher@york.ac.uk), Professor of Health Economics, Laura Bojke, PhD (laura.bojke@york.ac.uk), Research Fellow, Centre for Health Economics, University of York, Heslington, York YO10 5DD, UK
Jonathan C. Tosh, MSc (j.tosh@sheffield.ac.uk), Research Associate, School of Health and Related Research, University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, UK
CONFLICT OF INTEREST
For this work, L Longworth's institute has received a Methodology Grant from the NIHR Medical Research Council, and J Tosh and M Sculpher have received funding for their institute from the UK Medical Research Council. L Bojke does not report any potential conflicts of interest.
