The National Institute for Health and Clinical Excellence (NICE) was established in 1999 to provide clinicians with the tools required to provide high-quality health care (Reference Rawlins31). One of its core functions is to appraise healthcare interventions and recommend whether or not they be paid for by the National Health Service (NHS). To avoid a promising but unproven technology being excluded from use in the NHS, the Department of Health also allows NICE to recommend that an intervention be used only “as part of a well-designed program of research” (7).
Waiting for more information is sometimes better than making definitive but premature decisions (Reference Kesselheim, Fischer and Avorn12;Reference Sauerland and Maegele34;Reference Topol35). Because decisions based on insufficient evidence can have adverse effects on patients or inappropriately stifle a technology's development, the concept of paying for healthcare interventions only when provided in the context of ongoing research is receiving increasing attention in many countries (Reference Levin, Goeree and Sikich13;Reference Tunis and Pearson36;Reference Vinck, Neyt, Thiry, Louagie and Ramaekers37). The term “only in research” (OIR) is commonly used in the United Kingdom to describe these forms of approval; similar terms used internationally include “coverage with evidence development,” “coverage with study participation,” and “conditionally funded field evaluations.”
In the United Kingdom, governmental bodies as well as independent experts have suggested that NICE use the OIR option more frequently (Reference Chalmers4;11). NICE is also being increasingly asked to issue appraisals for drugs closer to launch, when the evidence base may still be immature, and OIR decisions may be more likely (8). Although it is NICE's responsibility to issue recommendations about whether health technologies should be used in the NHS or not, it has neither the funds nor the mandate to commission research. Furthermore, there are no formal arrangements between NICE, government, industry, research funding agencies, or the NHS to prioritize, commission, fund, and conduct research stemming from OIR recommendations (11). In practice, the lack of an identifiable pathway for OIR-related research has meant that OIR decisions have often been viewed as “no” decisions (Reference Raftery30). This situation has been identified as an area for improvement (Reference Cooksey6).
To the end of 2007, 43 of 455 (9.4 percent) NICE guidance documents have recommended that a health intervention be used only in research for one or more patient subgroups, with 61 separate OIR decisions (Table 1; Figure 1). In seven instances, research published after an OIR decision has resulted in NICE subsequently issuing a definitive “yes” decision (16–21, 24). More commonly, however, OIR decisions have not been followed by the research required to reduce uncertainty. In May 2008, NICE held an international workshop on managing uncertainty in health care in collaboration with the United States Agency for Healthcare Research and Quality (AHRQ). Details of how the workshop was conducted and what discussions took place have been published elsewhere (25). Toward the conclusion of the workshop, the facilitators identified several key issues related to OIR decision making; these are listed and contextualized in Table 2.
Table 1. Examples of “Only in Research” Decisions Issued by NICEa
aNICE, National Institute for Health and Clinical Excellence.
Figure 1. Only in research decisions between NICE's inception in 1999 and December 31, 2007.
Table 2. Key Issues Identified at the NICE-AHRQ Workshop on Managing Uncertainty in Health Careaa
aNICE, National Institute for Health and Clinical Excellence; AHRQ, United States Agency for Healthcare Research and Quality; OIR, only in research.
Building on the workshop and also the deliberations of NICE's Citizens Council in this area (26), we undertook a series of interviews with key stakeholders to obtain the views of stakeholders on the current arrangements for implementing OIR decisions and to investigate how the OIR pathway could be improved.
METHODS
The issues that were identified at the NICE-AHRQ workshop were used as a starting point in the development of a framework (Figure 2) to guide the interviews and data analysis (Reference Ritchie, Spencer, Bryman and Burgess33). The framework approach is a qualitative analytical method widely used in applied health policy research (Reference Pope, Ziebland, Mays, Pope and Mays29).
Figure 2. Conceptual model of decision-making process with potential solutions designed to improve the “only in research” pathway.
An initial sample of interviewees was selected purposively using input from NICE staff. We invited participants from a variety of different stakeholder groups, including industry, academia, government, the NHS, independent research funding agencies, and consultants. Consistent with standard qualitative research methodology, we did not calculate a sample size; rather, we continued interviewing individuals until no new concepts arose from the data analysis—a phenomenon referred to as saturation (Reference Higginbottom10).
We conducted eleven separate interviews with a total of thirteen interviewees during July and August 2008. Six interviewees had primary academic appointments; all six had extensive experience with clinical research and/or applied health economics, and most were providing advice to either government, NICE, or industry. Four interviewees worked in the pharmaceutical or medical device industries, one was involved in commissioning health services within the NHS, one interviewee held a senior position at a charitable agency, and another served as an external advisor to government.
Interviews were recorded and transcribed, and the framework was systematically applied to interview data using qualitative data analysis software (nVivo 8, QSR International). Analysis proceeded in conjunction with interviews, and the framework and interview guide were repeatedly modified to reflect the ongoing analysis. As a validity check, preliminary findings were discussed with both interviewees and NICE staff (Reference Mays and Pope15).
Ethical approval for this study was obtained from the London School of Hygiene and Tropical Medicine, and all interviewees provided informed consent.
RESULTS
The Use of the OIR Option in NICE's Decision Framework
All interviewees believed that the OIR option is both reasonable and useful. For example, a representative comment was that “[the OIR option is] sensible if used sensibly. . . [I]t seems a perfectly rational way forward as long as it's not used as a rationing tool. . . and it could get drugs into clinical use more quickly.”
Only one interviewee expressed any reservation, although even he acknowledged that “it is understandable that there would be occasions where it's a marginal call or NICE actually believes that a treatment does not represent good value for the NHS, and then I think the idea of [OIR] is useful.”
Another interviewee noted that OIR decisions should be accompanied by a clear explanation regarding why the available data were insufficient to make a definitive decision and what data would be required for NICE to be definitive. All interviewees except one were skeptical about using formal value of information analyses to determine whether NICE should issue an OIR decision (Reference Claxton and Sculpher5). Even the interviewee who endorsed the use of value of information analyses expressed concern that the technique was immature. On the other hand, most interviewees were enthusiastic about using expert groups to make assessments about whether research to reduce uncertainty would be both feasible and valuable.
Interpretation of the OIR Decision
Several interviewees noted that OIR decisions should be associated with more clarity. For example, one interviewee stated that “[T]here is opportunity for ambiguity so [in the] example that I've been involved in, it's not clear to me whether that was, to use North American parlance, coverage with evidence development, or whether it was an only in research requirement condition.”
Interviewees believed that the OIR option would likely be viewed as legitimate so long as two conditions were satisfied: first, that NICE decision-making processes in general were viewed as legitimate, and second, that research arising from OIR decisions be widely and equitably accessible. One interviewee, discussing the first point, stated that “at the moment there are a label of drugs in cancer which we either know or expect NICE to reject. Were NICE to issue any of those on an OIR, unless the OIR was designed to be so open that you could get a lot of patients into the trials, I think there may be a cynicism which people are saying, ‘Ah, NICE doesn't want to get the flak it deserves for rejecting an intervention, so it's ducking the issue by setting out an OIR.’ And then that would undermine the whole concept of an OIR, wouldn't it?”
Implementing OIR Decisions
Several interviewees pointed out the practical difficulties involved in making OIR decisions. For example, one interviewee stated: “When NICE says only in research, can they make it stick? And the answer is ‘No, they can't.’ Is there a study? Who should do the research? . . . I think to some extent it discredits NICE if it says only in research and then everybody ignores it.”
There was unanimous agreement that a formal process should be established to prioritize research needs arising from each OIR decision. If a determination were made that further health technology assessment research should be funded by the public sector, the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) program was identified by several interviewees as the most appropriate organization to coordinate further prioritization and research commissioning decisions. Several interviewees suggested that links between NICE and the NIHR HTA program be strengthened.
How Should Research Be Funded?
Two main issues arose when discussing research funding. The first was the difficulty in deciding when research should be funded by the commercial sponsor and when it should be funded by the public sector; the second related to how funds should be channeled within the public sector for publicly funded studies.
Although interviewees were unanimous that the public sector should pay for research when there was no relevant commercial sponsor and when the sponsor would have no commercial interest in pursuing additional research, interviewees were divided about who should pay for research that would benefit both the commercial sponsor and the public.
The current system of funding for publicly funded studies, where research costs (e.g., the costs of data collection and analysis, trial registration, publication, and so on) are paid for by the funding agency, but support costs (e.g., the costs of extra diagnostic tests and patient visits) and treatment costs (including the cost of the experimental treatment) are paid for by Primary Care Trusts (PCTs), was unanimously regarded as unsatisfactory. For example, one interviewee stated that “some of the divisions between service support and research costs [are] worrying. Say we do a special investigation for an outcome, something like an MRI scan . . . Now those MRI scans probably wouldn't have been done if the patient hadn't been in the study. And therefore they're being done explicitly for research reasons in my head. But that information will be fed back to the clinician managing the patient. And it could be argued that those results will influence the patient's subsequent care. So one argument would be that they're done explicitly for research therefore they're research costs. Another argument is . . . they are standard NHS investigations, they influence patient management, and therefore they're service support costs.”
Interviewees proposed two potentially complementary solutions for this problem: first, that all costs associated with research, including support and treatment costs, be funded centrally and separately from core PCT budgets; or second, that PCTs be required to participate in high-priority research arising from OIR decisions in the same manner that they are required to provide patients with drugs approved by NICE. Finally, there was a high degree of consensus that OIR-related research requiring public funds should be commissioned proactively rather than funded through a traditional, investigator-initiated, generic application process.
How Specific Should NICE Be about the Research Design?
There was general agreement among interviewees that NICE should not mandate how studies should be designed. Rather, most interviewees suggested that optimal research designs were likely to arise from an externally managed research prioritization and commissioning process that encouraged flexibility and investigator input. However, there was some disagreement about whether NICE should make the initial determination as to whether a randomized controlled trial would be necessary or whether an observational study would suffice. One interviewee who argued that NICE should make this determination suggested that NICE could distinguish “. . . between two [options] . . . One is where you say there . . . needs to be a formalized trial and therefore no patient will automatically get access to the drug because in the trial they will be randomized to placebo or a distinct treatment or whatever. [The other is] where you say there are some key elements of information that we need that we believe we can collect in a non-experimental context.”
Because NICE has sometimes recommended that registers be used to collect safety and efficacy data for all patients receiving a particular form of treatment, we also asked interviewees about this issue specifically (Reference Lyratzopoulos, Patrick and Campbell14). Interviewees expressed varying degrees of enthusiasm for the use of registries. For example, one interviewee stated “I'd be very attracted by registries if I was convinced they could work. But I'm not convinced they can actually work. There are numerous examples of where this kind of data has been misleading . . . I don't dismiss registries out of hand. If I could be convinced they'll give the answer it would be nice. They'd be a lot cheaper than trials.”
On the other hand, another interviewee stated “If ‘only in research’ promotes and encourages [registries], then I think it would be enormously helpful. Because at the moment what happens as we all know is that surgeons start new procedures and there's no control on it, and either the thing falls into disrepute and disappears, or it becomes established and then ten years later, we have to demonstrate that it's not a terribly useful procedure, and of course hopefully somewhere in the middle there are some procedures that turn out to be wonderful and thank goodness they were introduced, but we could be much cleverer with that, and I think if NICE could encourage the establishment of databases of some sort then that would be a really big contribution.”
Can OIR Decision Making Be Better Integrated with Other Areas of Healthcare Policy Decision Making?
Focusing on aspects of integration that might lessen the need for OIR decisions, we asked interviewees about the integration of NICE's decision making within the wider regulatory framework, as well as collaboration between NICE and industry and between NICE and the NIHR HTA program.
Many interviewees held strong views about the regulatory process, but there was no consensus about how it should be changed or even whether it should. Some interviewees argued that the regulatory process was outdated and that increased use of conditional approval would better serve the public interest (Reference Rawson, West and Appel32). For example, one interviewee stated: “There is a fundamental problem in the whole late stage development paradigm, which is defined by regulators. And that is, it doesn't provide the kind of crucial efficacy data, or indeed, the clarity of data even for safety, that you need in the modern world.”
Other interviewees believed that the information required by the regulator is very different from the information required by agencies like NICE and that the current regulatory framework served its purpose.
Most interviewees were supportive of greater collaboration between industry and NICE, particularly relating to NICE's decision to offer early advice to technology sponsors (22). Several interviewees stated that by providing early input into trial design, NICE was more likely to receive “fit-for-purpose” data, thus potentially reducing the need for OIR decisions. For example, one interviewee stated: “I think [collaboration between NICE and industry] is perfectly sensible. It gets back to much more of a partnership development between the NHS and the industry for new products, and again it's got to be a sensible thing to do.”
Two interviewees did point out that increased collaboration with industry might raise new problems, for example if changes to the standards of care resulted in NICE's advice being unhelpful to a technology sponsor, or if the independence of the Appraisal Committee were compromised in any way.
Summary of Results
Key findings, categorized by the degree of consensus among interviewees, are summarized below in Table 3.
Table 3. Key Findings from Interviewsa
aNICE, National Institute for Health and Clinical Excellence; NIHR, National Institute for Health Research; HTA, health technology assessment; OIR, only in research; PCT, Primary Care Trust.
Discussion
This report complements previous work NICE has done on the OIR issue (Reference Chalkidou, Hoy and Littlejohns2;Reference Chalkidou, Walley, Culyer, Littlejohns and Hoy3;26). Findings of particular interest, as well as those that could potentially lead to changes in policy, are discussed in greater detail below.
The Use of OIR Option in NICE's Decision Framework
Interviewees unanimously agreed that using the OIR option was preferable to making definitive but premature decisions. Formal methods developed by health economists during the past 10 years, often called “value of information” analyses, provide an attractive theoretical foundation for issuing OIR decisions in appropriate situations (Reference Claxton and Sculpher5;Reference Palmer and Smith28). Interviewees noted however that these methods may be difficult to apply in specific cases and suggested that NICE develop a formal process using experts to help decide whether research in a particular area would be practical and likely to reduce uncertainty.
Details Accompanying the OIR Decision
Several interviewees noted that, for an OIR decision to be helpful, it should be accompanied by a clear statement regarding what data are required to reduce uncertainty. In most cases, this statement is likely to suggest that additional data are needed to establish superior effectiveness compared with the established standard of care (23). Where the nature of the evidence required by NICE is clear, it would also be appropriate for NICE's Appraisal Committee to suggest the type of study that would be most appropriate. Health technology assessment agencies should also consider establishing policies so that OIR decisions are accompanied by prominent, structured statements that clearly articulate what data are needed to reduce uncertainty (Reference Brown, Brunnhuber and Chalkidou1).
Early Discussions with Commercial Sponsors
The results from this study provide additional support for early discussions between health technology assessment agencies and technology sponsors (22). If these discussions achieve their intended aim, the relevance of studies available to organizations like NICE would be higher and the likelihood of an OIR decision would be reduced. In the UK, these consultations have already received widespread support (Reference Cooksey6;22).
Prioritizing and Implementing OIR Decisions
The results of this study highlight the importance of developing a transparent pathway to prioritize and implement OIR decisions. Our interviewees suggested that, in the United Kingdom, this process would be facilitated by NICE and the NIHR HTA program strengthening their relationship (Reference Chalkidou, Walley, Culyer, Littlejohns and Hoy3;27), with the two organizations initiating discussions as soon as possible after the meeting at which an Appraisal Committee undertakes its first review of the topic and indicates that it is likely to make an OIR recommendation. As with other NICE processes, stakeholders should be involved and the results of the discussion subject to public consultation. If through this process, research arising from an OIR decision is deemed to be a high priority for public funding, it could be fast-tracked through the NIHR HTA program's standard commissioning route, or alternatively, by means of a dedicated pathway. In the long-term, a parallel process where the NIHR HTA program routinely provides expert advice to NICE on whether OIR decisions are practical before these decisions are made may be preferable.
Research Design
Although randomized controlled trials are generally considered to be the most appropriate method to generate evidence regarding relative treatment effects, evidence from nonrandomized experimental and observational studies is necessary, not only where RCT evidence is unavailable but also where there are concerns regarding generalizability (23). When asked to discuss the “only in research” issue, NICE's Citizen Council suggested that NICE should not only “define the questions it wants answered through research” but also that it should “prescribe the methodology to be used” (26). In contrast, the individuals interviewed for this study suggested that, although NICE might indicate its preference, the research commissioner (in collaboration with individual researchers) should ultimately decide what study design would be most appropriate. This process however must involve sufficient dialogue to ensure that the research findings meet the needs of the decision-making body.
Research Funding
Ensuring that research funding arrangements do not discourage the participation of healthcare providers in OIR-related research is of vital importance. The current situation in the United Kingdom, where costs associated with research studies are divided into three categories (research costs, which are paid for by the research funding agency, and support costs and treatment costs, which are generally paid for by the care provider) was widely viewed by interviewees as a barrier to research. Notably, the fact that a single governmental department has the mandate and resources necessary to prioritize, commission, and fund all aspects of the required research has contributed to the success of an OIR-like framework used for nondrug healthcare interventions in Ontario (Reference Goeree and Levin9;Reference Levin, Goeree and Sikich13). This is in contrast to the United Kingdom, where NICE, the HTA program, and the Department of Health and Primary Care Trusts (PCTs) have different roles and funding responsibilities.
In the United Kingdom, funds for all aspects of OIR-related research could be provided to PCTs separately from core funding, either directly from the Department of Health or indirectly by means of the research funding agency. Another option would be to insist on PCTs participating in high-priority OIR-related research using their existing budgets. The first approach would provide a positive incentive to PCTs to participate in research. The second approach, in contrast, would be difficult to enforce and might be less likely to produce a primary care culture conducive to performing clinical research.
Limitations
Several important limitations of this study should be noted. First, our study was limited in scope. We intentionally chose not to discuss ethical issues relating to OIR decisions, and only briefly discussed the methodological and procedural criteria used to reach OIR decisions. Second, its findings may not be fully generalizable to non-UK jurisdictions. Third, despite the heterogeneous sample of interviewees and our achievement of saturation in determining sample size, it is possible that other viewpoints and issues would have arisen had more individuals been interviewed. For example, we did not include lay individuals or patient representatives in our study sample. However, NICE's Citizens Council has discussed the OIR issue extensively (26), and ongoing discussions with NICE staff during data collection confirmed that saturation was achieved.
POLICY IMPLICATIONS
The conceptual basis for OIR is well established both on theoretical and historical grounds. The key challenges and unanswered questions that remain are largely practical in nature: When and how exactly should an OIR decision be issued? How should research needs arising from an OIR decision be prioritized, commissioned, and funded? Who should design these studies? How can the OIR mechanism be designed in a way to maximize legitimacy and public acceptance? How can the process be developed so that relevant research findings are fed back into the health technology assessment process in a timely manner? Although answers to these questions will vary, not only between different healthcare systems, but also for different healthcare interventions within a single healthcare system, there are likely to be common principles that health technology assessment agencies can use as a basis to address these issues.
The findings of this study suggest that health technology assessment agencies should consider using an OIR option when faced with uncertainty, and suggest several ways in which this pathway could be optimized. The main implications of our study are that health technology assessment agencies should be explicit about the data required to reduce uncertainty, they should consider offering early advice to technology sponsors in an effort to preempt OIR decisions, they should establish a formal process in collaboration with research commissioners to prioritize and implement OIR decisions, and the funding system for OIR-related research should be designed in a way that does not discourage OIR-related research.
CONTACT INFORMATION
Irfan A. Dhalla, MD, MSc, FRCPC (dhallai@smh.toronto.on.ca), Lecturer, Medicine and Health Policy, Management and Evaluation, University of Toronto, Clinician Scientist, Keenan Research Centre of the Li Ka Shing Knowledge Institute, St. Michael's Hospital, 30 Bond Street, Toronto, Ontario M5B 1W8, Canada
Sarah Garner, PhD, BPharm, MRPharmS (sarah.garner@nice.org.uk), Associate Director, Research and Development, Kalipso Chalkidou, MD, PhD (kalipso.chalkidou@nice.org.uk), Acting Director, Policy Consulting, Peter Littlejohns, MD, FRCP (peter.littlejohns@nice.org.uk), Director, Clinical and Public Health, National Institute for Health and Clinical Excellence, 71 High Holborn, London, WC1V 6NA, UK
