Health technology assessment (HTA) is a multidisciplinary field of research created in the 1970s to support the process of decision making in healthcare at the policy level (Reference Hannon10). Whereas in the beginning, HTA basically relied on systematic reviews and analysis of published literature to assess the short- and long-term consequences of the application of a health technology (for example, societal, economic, ethical, legal), HTA methods have evolved to adapt themselves to the demands and expectations of decision makers. In the past decade, the exponential growth in technological innovations in an scenario of limited budgets and more informed patients that demand rapid access to promising technologies, has prompted the interest in research frameworks that can provide decision makers with additional information on health technologies, while providing timely access to the new treatments (Reference Fronsdal, Facey and Klemp8;Reference Trueman, Grainger and Downs21).
To this end, many countries, including Spain, have implemented different policy frameworks to facilitate the controlled use of innovations, restricting access to specific conditions (selected centers, specific indications, patient subgroups, etc.) and linking reimbursement to targeted data collection. These frameworks, destined to generate evidence to inform decision making, are frequently defined as coverage with evidence development (CED), terminology coined by the Center for Medicare and Medicaid Services in the United States, even though they receive other terms internationally: “access with evidence generation systems (AEG)” within the EUnetHTA project, “field evaluations” in Canada (Ontario), “still in Research “in France “interim funding” in Australia, “only in Research” in the United Kingdom and “monitored use” in Spain (Reference Carbonneil, Quentin and Lee-Robin3;Reference Dhalla, Garner, Chalkidou and Littlejohns5;Reference Levin, Goeree and Levine12;13).
There are many different examples of successful implementation of CED schemes, the majority focused on providing additional evidence on nonpharmaceuticals when there is uncertainty about safety, effectiveness, or cost-effectiveness at the initial coverage decision state. An overview of international experiences within the EUnetHTA project has identified twelve countries that have implemented AEG mechanisms for medical and surgical procedures (Reference Carbonneil, Quentin and Lee-Robin3). For example, this approach has been used to solve uncertainties for PET scanning and endovascular treatment of abdominal aneurysms in the United States (Medicare and Medicaid), Australia, Canada (Ontario), and Spain.
While many stakeholders agree that CED frameworks that restrict coverage to controlled environments can be useful to address research gaps in HTA, important concerns have been raised about the applicability of these approaches. It is argued that uncertainties are always present when new technologies are launched and that the benefits and harms of some technologies, especially very innovative or costly ones, can greatly depend on the condition that they are appropriately used once they are diffused in real world settings (Reference Eisenberg7). In addition, it has been proposed that, if conditional coverage is initiated after a technology has already diffused within the health system, some patients who are candidates for the procedure may be denied it if they are not enrolled in the trial; the same is true for patients that are not attended in the assigned centers. Another drawback to CED is that medical procedures frequently evolve during the research process, to the point that the trial's findings might be of diminished relevance when the results are available (Reference Goodman9).
In line with these concerns, many international institutions are beginning to consider that collecting data after the technology is included in the reimbursement list might be the only way to evaluate the real impact in the health system, without denying therapy to patients. However, within the European Union (EU), the observation of new technologies after coverage has practically only been considered a prerequisite for approval for pharmaceuticals and most studies are not focused on supporting long-term funding decision making before widespread adoption and only address safety concerns (6).
Although important investments have been made in observational studies to measure postcoverage outcomes, research has been mainly focused at comparing the success rates of different treatment schemes and has not been specifically designed to address material uncertainties around new technologies and thus, help inform future decisions on diffusion or application. For example, in the UK National Health System, Patient Reported Outcome Measures (PROMS) have been implemented to provide information on the quality of care of four conditions: hip and knee replacements, varicose vein surgeries and groin hernia surgery. The health status information is collected from a questionnaire administered to patients before and after the intervention and provides only an indication of the outcomes and quality of care of different technical approaches (20). Within the United States, the Effective Healthcare Program has supported great investment in postmarketing efficiency research, especially in the comparative effectiveness area, but the research has also been focused at costly conditions that present great variability and has not been based on prioritized HTA needs or predefined methodological standards (Reference Norris, Atkins and Bruening14).
To date, successful examples of CED's frameworks that were sponsored to address the performance and use of technologies in real world settings come mainly from Ontario's Field Evaluation Program (Reference Levin, Goeree and Levine12). In this Canadian province, field evaluations address uncertainty regarding efficacy, effectiveness, safety, or applicability. The few studies that have assessed the performance of technologies in real world conditions demonstrate the value for providing a sound basis for long-term funding decisions but state the need for rigorous methodology to increase acceptability.
The present article aims to further stimulate the discussion on systematic post-introduction observation of health technologies. The study outlines the reasons why it is important to assess technologies after coverage and presents a summary of the guide commissioned by the Spanish Ministry of Health, Social Policy and Equality to the Galician HTA agency for the prioritization and implementation of systematic post-introduction observation of health technologies in Spain.
RATIONALE FOR THE POST-INTRODUCTION OBSERVATION OF NEW HEALTH TECHNOLOGIES
Identify Safety and Effectiveness Problems That Only Appear in Real Practice Conditions
When applying new healthcare technologies under normal conditions of use, there is the possibility that effectiveness or safety problems unforeseen in preliminary studies might appear. Even if adequately designed, preapproval studies are frequently conditioned to restrictive inclusion/exclusion criteria, have a low sample size, are restricted to specific application protocols, and are undertaken at more specialized healthcare facilities, making them inadequate to provide generalizable information on effectiveness or safety (Reference Hannon10). In fact, a recent systematic review has shown that as few as 20 percent of the relevant patients have been included in pharmaceutical randomized clinical trials (RCT) (Reference Botting, Reilly and Harris2). For medical devices and procedures, these values could be even worse due to the fact that, unlike drugs, they are not required to demonstrate its clinical efficacy in RCT. Preapproval assessments are frequently based on case series conducted by experienced clinicians on small groups of patients and in modified conditions, underestimating serious harm or long-term adverse events.
Assess Effectiveness and Safety in Comprehensive Clinical Scenarios and Specific Subgroups
Sometimes preliminary investigations are not frequently aimed to assess the effectiveness or safety in comprehensive clinical scenarios (nonspecialized centers, less resources) or specific subgroups (severe cases, patients with co-morbidities, aged, pregnant women, etc.). Most preliminary trials allege excluding sensitive subgroups because of ethical concerns, or when they include them, the sample size is usually insufficient (Reference Sorensen, Lash and Rodman19). To obtain high quality evidence before the approval of a new technology might require very large randomized trials including a group representative of all patients to be treated in clinical practice and similar scenarios. Within Europe premarketing clinical trials are only mandatory for pharmaceutical products. For medical devices and procedures, whose implementation requires sometimes costly infrastructure and acquiring training skills, the premarketing studies are only aimed at demonstrating safety and performance (6). Setting up randomized clinical trials is recognized to be very costly, timely, and argued that it could greatly delay the introduction of innovations. As a result, when a technology is approved, it is frequently very difficult to identify the subsets of patients in whom the technology might not be effective or the population at higher risk for adverse events. Collecting disease specific and generic data from a large population cohort of patients treated in real world practice can allow for comparing outcomes in different subgroups and help decision making regarding appropriate indications and best clinical management strategies.
Identify Off-Label Use
The use of new technologies outside its approved indications, which is frequently referred to as off-label use of technologies, is spreading in many healthcare systems. This phenomenon is difficult to manage from a policy point of view and also from an evidence-based one. Data from the American National Cardiovascular Data Registry reveals that off-label use of drug-eluting stents occurs in 24.1 percent of the procedures (Reference Rao, Shaw and Brindis16). This practice occurs on a case by case basis, and frequently the indications and outcomes of the patients are unknown. Detecting off-label use is relevant because it really amounts to investigation without consent. The use of new technologies in indications different than those authorized should be identified to assess the real value of those indications. There is also off-label deviation from evidence-based guidelines, which is not so much deviation from policy authorization but a guideline deviation. A recent study has shown that in the United States, 22.5 percent of implantable cardioverter defibrillator did not meet evidence-based criteria for implantation (Reference Al-Khatib, Hellkamp and Curtis1).
Assess the Degree of Adoption and Usage of Health Technologies
Even though it is generally accepted that, once approved for reimbursement, new technologies will be gradually adopted within the healthcare system, adoption can vary substantially and create important inequities. The adoption, implementation, and assimilation of health technologies can be greatly influenced by financial constraints, existence of organizational, structural or technical limitations, or on the acceptability by clinicians or patients (Reference Robert, Greenhalgh, Macfarlane and Peacock17). The implementation of some new technologies can produce major changes in the flow of patients and patient's management. Such is the case of many complex interventional procedures, which can require sophisticated hospital resources, high staff requirements, significant hospital stay, and so on. It is likely that diffusion at 1st-level hospitals may not be the same as at 2nd- or 3rd-level hospitals, in rural or urban areas, and this could give rise to important inequities, with the consequent variability in patient outcomes. Tracking the place of residence of the treated patients through a monitoring claims database or by other means (electronic record, medical history) can serve to measure if observed geographic distribution differs from that expected and identify variations. Noting variations can detect accessibility problems and be a factor of centers of excellence.
The different degree of adoption of healthcare technologies can have different causes, that is, the different willingness or resistance of individual health professionals or patients to use innovative or invasive technologies, the desire of healthcare facilities to test new technologies, or even health organizations wishing to promote their visibility among the electors. The consequences can result in a great variability in patient outcomes and inefficient use of resources (Reference Whitted25), including overuse or inefficient expenditure. Because all health organizations have limited budgets, identifying these inefficiencies is very relevant. The use of any unnecessary resource suggests fewer resources for other effective healthcare.
To sum up, post-introduction assessment of new health technologies can be essential for identification and assessment of problems concerning implementation, accessibility, acceptability, and adequacy of use, and can provide reliable and comprehensive information on the real costs and use of resources that are derived from the application of health technologies in real settings. Likewise, it can enable for the detection and assessment of relatively rare adverse effects or problems of effectiveness that are only observable with large sample sizes or in specific populations. Depending on the clinical scenario and the type of intervention to be evaluated, the information can be used to refine clinical indications, patient management, or used to design organizational or structural interventions aimed at optimizing technology usage and allocation of available resources within a healthcare system. Table 1 summarizes the main goals of post-introduction assessment of new technologies.
Table 1. Main Goals of Post-introduction Observation of New Health Technologies
METHODS
The current study was developed to provide guidance for implementation of post-introduction observation within the framework of the Spanish National Healthcare System (Reference Varela Lema, Ruano Raviña and Cerdá Mota24), which is a single-payer decentralized system that provides universal coverage for approved technology indications. The guideline is focused on medical procedures and devices, because coverage with evidence development is not envisioned for pharmaceuticals in Spain.
The guideline was performed on the results of a bibliographic search and on the opinions of a group of ten Spanish HTA experts coming from six different regional HTA units. The review identified the main aspects to be covered in the guideline and provided information on other experiences and opinions in this field. A workshop was held to discuss key papers and personal experiences and address key methodological aspects. During the workshop, the working group agreed upon the main requirements for implementing a post-introduction observation system and established the methodology for addressing the different recommendations, which are briefly discussed below.
RESULTS
Prioritization of New Technologies Susceptible of Post-introduction Observation
Like other initiatives that require data collection, post-introduction observation of health technologies requires for a great investment in time, resources, and implication from different bodies. For this reason, it is generally assumed that the decision to observe a new technology must be prioritized, taking into account the capacity of the health system and the value of collecting additional data for decision making (Reference Claxton and Sculpher4;Reference Trueman, Grainger and Downs21). At present, even though there is an agreement that observational data on health outcomes can contribute to HTA, the criteria for determining when this type of assessment should be carried out remains unclear. In part, the criteria can depend on the purpose and intended actions from the observation activities, and these can vary in different healthcare settings and with the nature of the interventions.
In our context, post-introduction observation is envisioned in the framework of CED and is aimed at providing information regarding uncertainties that can appear when devices or procedures are introduced into the public health service's portfolio. The tool developed as part of this project aims to help decision makers to discern which technologies merit for observation. It is a quantitative tool that includes a list of weighted criteria and allows for scoring and comparing up to fifty technologies.
The prioritization tool was developed in two phases. The first phase consisted in reviewing the literature to develop a prioritization criteria proposal. This proposal was analyzed by the working group and agreed upon through a consultation process. In the second phase, a modified Delphi method was used to select and weigh final selection criteria. The prioritization panel involved thirty-six national experts representative of the different sectors involved in the adoption and use of the technologies (policy makers/hospital managers, clinicians, and patients). Each participant ranked the fifteen criteria initially proposed from 1 to 9 and weighted each of the four proposed domains to a total of 100 points. Those criteria that scored 6 or lower were excluded. The final list includes fourteen prioritization criteria that cover different factors that determine the relevancy of additional data collection. Among these, factors that might confound preliminary results and create uncertainty regarding the application of the technology in real world settings: vulnerability of target population, different expectations of use, innovativeness, and undetected potential adverse effects. The prioritization criteria and the weightings are detailed in Table 2.
Table 2. List of Selected Prioritization Criteria Grouped by Domains and Information Requirements for Prioritization
The developed tool, named PriTec (http://www.pritectools.com), allows the user to score the prioritization criteria from 1 to 9, automatically calculating the score for each domain and the total score for each technology, furnishing the absolute and weighted scores. The tool, conceived to be used along with supporting scientific evidence, has been tested by nine health professionals in a workshop and has revealed a high reliability for prioritizing health technologies deemed relevant of further observation (overall intraclass correlation coefficient value of 0.95 [95 percent confidence interval, 0.89–0.99]). While the value of this tool is currently being further explored, including assessment in a wider set of stakeholders (policy makers, hospital decision makers, clinicians), preliminary results suggest that, once refined, it could serve to provide an objective structure for making recommendations in HTA assessment reports or can be used by decision makers as a starting point for discussing the relevancy of a post-introduction study in our context. Even though the value of the criteria and the weightings might differ in other healthcare setting, they can clearly serve as a starting point for further discussions and investigations regarding final criteria to establish priorities in post-introduction observation.
Protocol Development and Outcome Indicators
The implementation of observational studies requires for the development of a common study protocol that defines the study design, the relevant outcome indicators to be investigated and the standards deemed acceptable or desirable for each new technology targeted for assessment. In accordance with international HTA experts (Reference Fronsdal, Facey and Klemp8;Reference Hutton, Trueman and Henshall11;Reference O'Malley, Selby and Jordan15;Reference Trueman, Grainger and Downs21) and taking into account the results of an interview with decision makers and clinicians from the Galician Public Health System (Reference Varela Lema, Atienza Merino and López García23), there was a common agreement that protocols must be drawn up taking into account available evidence and also the opinion of the different stakeholders involved in the usage of the new technology within the healthcare system. The different stakeholders (HTA experts, clinicians, decision makers, patients) should be implicated from the beginning and should be involved in establishment of data requirements, data collection methods, follow-up schedule, and outcome indicators that are considered feasible, clinically meaningful, and relevant to the healthcare system for decision making. They should also collaborate in the establishment of reference standards to identify and evaluate important deviations from the expected results of the technologies being observed. Implicating all key parties increases perception of relevancy and facilitates the adoption of policy measures. Many standard HTAs conduct epidemiologic and economic modeling which provides a basis for comparing actual versus predicted diffusion, and this can be a valuable aid on the quantification of reference standards.
The success of the observation experience resides on the value of the information but also on the acceptance and feasibility of the study. Based on expert opinion and considerations derived from national and international monitoring experiences (Reference Carbonneil, Quentin and Lee-Robin3;Reference O'Malley, Selby and Jordan15), it was agreed that post-introduction observational studies must comply with the following premises to increase compliance and avoid follow-up losses: (i) only data related to the key questions to be explored should be collected (minimum data-set); (ii) data recorded should be, as far as possible, part of the data recorded in medical records or considered sufficiently important for clinician's or stakeholders; (iii) the study should not lead to an increase in the number of diagnostic procedures or tests performed on the patient; (iv) whenever possible, the follow-up contacts should be in agreement with the routine check-ups; (v) the follow-up should be as short as possible, even though sufficient for obtaining an adequate number of patients in each target study-subgroup; (vi) management and analyses of results should be undertaken by specialized reference units, for example HTA agencies, with no financial self-interests and adequate training and skills; (vii) a Web-based tool with all the outcome variables should be used for data management; (viii) results generated should be continuously fed back to the user. Figure 1 outlines the framework proposed for implementation of post-introduction observation in Spain.
Figure 1. Framework for post-introduction observation of new health technologies.
DISCUSSION
Even if preliminary requirements are met, the implementation of observation studies is acknowledged to be difficult. The results of a systematic review on data collection instruments, undertaken as part of this project, support the idea that, at present, the most appropriate methodology for conducting post-introduction observation of new technologies is the use of prospective clinical registries. However, clinical registries that rely on clinician's participation are known to present important losses of patient data and adherence problems. Even though these limitations can be minimized collecting only relevant information and adjusting the follow-up contacts to the programmed medical visits, involving health professionals in data collection can be a difficult task. They have to dedicate time and resources, which are often scarce. In the near future, the use of electronic medical records with a linkage mechanism could clearly simplify this task but at present, many countries are still in the implementation stage and need to rely on specific questionnaires to collect patient information.
Different experiences suggest that clinician's participation can be higher if there is a policy framework that contemplates the use of data in the revised decision process (Reference Carbonneil, Quentin and Lee-Robin3;Reference Trueman, Grainger and Downs21). In our context, post-introduction observation pilot schemes implemented for various technologies, among them “sacral root stimulation for the treatment of fecal incontinence” and “percutaneous aortic valve replacement” serve to show that data collection can be effectively achieved when it is a prerequisite for future reimbursement of the technology (internal documents). Table 3 provides a list of the technologies that have been selected for post-introduction observation under this scheme and provides the main outcomes derived from these studies.
Table 3. List of Post-introduction Observation Processes Implemented in Spain (Galician Region)
Like the authors of the SWISSspine registry, our group believes that having an expert institution responsible for study set up, implementation, and follow-up is a key factor to ensure success (Reference Schluessmann, Diel and Aghayev18). Finding practical mechanisms to fund post-introduction observational studies is another problem. There is at present a great debate as to who should be responsible for financing additional research activities (Reference Trueman, Grainger and Downs21). Dedicated financing mechanisms for data collection might greatly depend on payment structures, which are specific for each country. In Spain, like in Ontario, the Department of Health is responsible for prioritizing conditional coverage research and commissioning mandatory clinical registries. HTA agencies are designated as the organism responsible for controlling the research design, establishing the outcome indicators and analyzing the results. In other countries, where there is not a single payer, the situation can be more complicated. In this sense, much can be applied from pharmaceuticals in terms of regulations and cost-sharing arrangements. For example, under the fee for service structure, the payment could be reduced, but this question is open for debate. Specific financing mechanisms should be fit for purpose and be adapted to the specific health scenario.
The difficulty to provide good quality evidence for comparison of different new treatment options have also been strong arguments against observational registries. However, within the framework of comparative effectiveness research, observational cohort studies comparing different indications have served to yield important information not evidenced in randomized clinical trials (Reference Turner, Hollingworth, Comstock, Downs and Deyo22). The strength of these studies lies in the large number of patients included, which avoids the problem of performance bias that can occur if patients are recruited from selected surgeons and centers. However, these large scale programs require extensive logistical support. In Spain, HTA agencies have effectively accomplished this mission, demonstrating the feasibility of these schemes.
CONCLUSIONS
At present, uncertainties still remain regarding the observation of new technologies after their introduction in standard clinical settings. The recommendations presented in this study were partly based on Spanish experiences so the proposal might not be totally applicable to other countries with different payment structures or conditional coverage schemes. In our country, CED results can be used to confine reimbursed indications if inappropriate care is identified and this has been recognized to produce important savings in the healthcare systems. In other settings, the value of the information could be different and other mechanism might be more appropriate for surveillance. Nevertheless, with independence of the perspective, we believe that methodological key questions regarding the establishment of post-introduction observation systems can be applicable to many organizations worldwide. Even though the study is not focused on pharmaceuticals, much can probably be applied to these in countries where conditional coverage policy structures oversee both pharmaceuticals and nonpharmaceuticals. More investigation should be done in this area to compare experiences among different countries and ascertain the real value of these types of initiatives.
POLICY IMPLICATIONS
Once introduced in clinical practice, health technologies are frequently neglected and no further studies are performed to ascertain if they have achieved their maximum value or the greatest level of benefit. In an ideal situation, it seems clear that technologies that show great uncertainty should be observed. There is evidence (13) that these studies can provide valuable information to decision makers and to the health community, information that would be otherwise very difficult to obtain. However, implementing these initiatives is a challenge for the majority of healthcare systems. It is recognized that setting up these studies requires specific policy frameworks, commitment from different stakeholders, and dedicated funding. Countries that aim at adopting such schemes need to consider the value of the information in their healthcare organizations.
CONTACT INFORMATION
Alberto Ruano-Ravina, MSc, MPH, PhD, Professor of Preventive Medicine and Public Health, University of Santiago de Compostela, Spain, CIBER de Epidemiología y Salud Pública, CIBERESP. School of Medicine, C/ San Francisco s/n, Santiago de Compostela CP 15782, Spain. alberto.ruano@usc.es
ENRIQUE SOTO PEDRE MD, MPH, MSc (epid), A Coruña University Hospital Foundation, Director, e-mail: Enrique.soto.pedre@sergas.es
Nora Ibargoyen-Roteta PhD, Researcher, Osteba, Osasun Teknologien Ebaluazioaren Zerbitzua/Basque Office for HTA Osasun eta Kontsumo Saila - Health and Consumer Affairs Department Olaguibel, 38, 01004-Vitoria-Gasteiz Eusko Jaurlaritza - Basque Government
Iñaki Gutierrez, BSc, MSc, M in Bioethics and PhD, Osteba, Osasun Teknologien Ebaluazioaren Zerbitzua/Basque Office for HTA Osasun eta Kontsumo Saila - Health and Consumer Affairs Department Olaguibel, 38, 01004-Vitoria-Gasteiz Eusko Jaurlaritza - Basque Government
Leonor Varela Lema (MPH, PhD), Researcher, Galician Health Technology Assessment Agency, Galician Department of Health, Edificio Administrativo San Lázaro, s/n 15781 - Santiago de Compostela + 34 881 54 86 09 + 34 881 54 28 54
Dr. Iñaki Imaz, MD, PhD, MPH, AETS (Agency for Healthcare Technology Assessment), ISCIII (Carlos III Institute for Health), 5 Monforte de Lemos, 28029, Madrid, Spain, Tel: + 34 91 8222088
CONFLICT OF INTEREST
Leonor Varela-Lema and Alberto Ruano-Ravina report their institution has received a grant from Institute of Health Carlos III, Spanish Ministry of Health and Consumer Affairs. Laura Sampietro-Colom and her institution have received funding for consultancy from GSK Roryzon-Reig Jofre and PriceSpective. The other authors report they have no potential conflicts of interest.
