Medicines regulatory agencies assess a new pharmaceutical on the basis of efficacy, safety and quality data and recommend the granting of a marketing authorization when there is a favorable balance of the pros and cons of the clinical outcomes for patients (also referred to as the benefit-risk balance) (Reference Eichler, Bloechl-Daum and Abadie1). Due to continuous rising costs in health care and budget restraints third-party payers require that new, mostly expensive pharmaceuticals also have a substantial added value compared with treatments that are already available (Reference Eichler, Bloechl-Daum and Abadie1;Reference Franken, Nilsson, Sandmann, de Boer and Koopmanschap2). One of the most commonly claimed values is an added clinical benefit and/or better safety profile. Comparing the clinical benefits and harms of a (new) technology with one or more (older) technologies used for the same condition is commonly referred to as comparative effectiveness in the United Stated (US), or relative effectiveness (RE) in Europe. In many European countries RE assessments of pharmaceuticals are performed as part of the reimbursement decision making process (Reference Kleijnen, George and Goulden3). These RE assessments need to be performed in a limited timeframe (90 days for pricing or reimbursement decision or 180 days for pricing and reimbursement decision) to achieve fast access for patients to new pharmaceuticals as was laid out in the Transparency Directive (Directive 89/105/EEC) (Reference Kleijnen, Goettsch and d’Andon4).
There is general consensus that the decision-making process on reimbursement decisions should be undertaken within national or local contexts in European Union Member States. However, there are potential efficiencies to be gained from enhanced collaboration around the collection of evidence underpinning these decisions. Increased sharing of information (e.g., methods, data requirements, and results) across jurisdictions may increase the quality and consistency of RE assessments in Europe. This may prevent duplication of information in various countries and save resources accordingly (Reference Kleijnen, George and Goulden3). To achieve this, there is a need for tools which allow standardized production and transparent reporting of RE information. Kleijnen et al. (Reference Kleijnen, George and Goulden3) concluded in a previous study that there are more similarities than differences in the methodology used for RE assessments in European jurisdictions indicating that a standardized, cross-border production of RE assessments may be possible.
The HTA Core Model® has been developed by the member organizations of the European Network of Health Technology Assessment (EUnetHTA) as part of EUnetHTA Joint Action 1 (JA1), a 3-year project (2010–2012) that was initiated by the European Commission to stimulate sustainable HTA collaboration in Europe. The aim of the Model is to facilitate the production and reporting of shareable pieces of HTA information. The HTA Core Model® consists of a set of generic questions (i.e., assessment elements) grouped in nine domains (see Figure 1) (Reference Lampe, Mäkelä and Garrido5). For a given assessment, assessors select questions from the model that they consider relevant for their topic and formulate them into research questions specific to their assessment. The model also contains methodological guidance for answering the questions and provides a structure for reporting. The idea of splitting HTA in research questions and their answers (including the methods for producing the answers) is that for national report production it is probably easier to adapt information from standardized pieces of information than from a traditional foreign national report. One can easily find and select the pieces of information required for national purposes and evaluate the validity of the information due to the structured and detailed reporting. As assessment questions may be specific to different types of technologies, specialized applications were already available in 2009 for medical and surgical interventions and diagnostic tests. The HTA bodies involved in developing a tool for standardized production and reporting of RE information of pharmaceuticals decided to adapt the framework of the HTA Core Model® to suit the expectation and requirements of RE assessments of pharmaceuticals.
Figure 1. Development of the HTA Core Model® for Rapid Relative Effectiveness Assessment of Pharmaceuticals (model for rapid RE assessment).
There are some essential differences between the traditional approach of the HTA Core Model® and doing a rapid RE assessment of pharmaceuticals. The main reason for these differences is that a RE assessment has a narrower scope than a traditional HTA. A RE assessment is a specific element of HTA that focuses on the clinical benefits and harms of the intervention, whereas HTA is broader and can also include other aspects, such as ethical, cost, and cost-effectiveness considerations (Reference Kleijnen, George and Goulden3). Second, as a rapid RE assessment is usually conducted just after the pharmaceutical has received market authorization, the data available for doing the assessment is often limited. Finally, the restricted timelines (90 or 180 days) for achieving fast patient access to new pharmaceuticals impose working restrictions on what is achievable within these timelines. Based on these differences in approach, a new application of the HTA Core Model®, the HTA Core Model® for Rapid Relative Effectiveness Assessment of Pharmaceuticals, was developed. In addition to the RE assessment model, nine methodological guidelines were developed for conducting a rapid RE assessment of pharmaceuticals. Some of these guidelines are discussed in detail elsewhere in this journal (6).
The aim of this article is to describe the development of the HTA Core Model® for Rapid Relative Effectiveness Assessment of Pharmaceuticals, hereafter referred to as the RE assessment Model.
METHODS
The development of the RE assessment Model took place in three phases (see Figure 1) and was led by a coordination team at the Dutch Health Care Institute (ZIN, formerly know as CVZ).
Phase 1
From May 2010 to March 2011 the first version (v1.0) of the RE assessment Model was developed. The “Cost and Economic Considerations Domain” was explicitly excluded at the beginning of the project based on the recommendations of the High Level Pharmaceuticals Forum (HLPF) that health economic assessments of costs and benefits should not be the primary focus of the European collaboration (7). The HLPF was a high-level ministerial platform for discussion between European Union (EU) Member States, EU institutions, industry, healthcare professionals, patients and insurance funds between 2005 and 2008. A specific working group on RE aimed at supporting Member States to apply RE systems to allow containment of pharmaceutical costs as well as a fair reward for innovation. The definition from the RE working group on RE was adopted: the extent to which an intervention does more good than harm compared with one or more intervention alternatives for achieving the desired results when provided under the usual circumstances of healthcare practice (8). The first version of the RE assessment Model was produced by 8 working groups called Domain teams that included individuals from 18 HTA organizations from 12 European countries. Each team focused on one domain and the roles of the individuals in a team were divided into authors and reviewers. The authors used the existing text of the HTA Core Model® v1.0 for Medical & Surgical Interventions as well as draft versions of the HTA Core Model® v1.0 for Screening Technologies as base text and had the task of updating the text and to consider adding specific elements and methodological guidance for pharmaceuticals. Reviewers commented on the draft versions of authors’ work. Subsequently, all domains were integrated into one document which was sent to all work package 5 (WP5)/JA1 organizations for comments in January 2011. The comments were processed by the authors of the domains resulting in the first version (v1.0) of the RE assessment Model (March 2011).
Phase 2
In phase 2 (April 2011 – January 2012), a pilot assessment was conducted to test the first version of the RE assessment Model and Guidelines for rapid RE assessment to collect feedback on the content and feasibility of the tools. The topic of the pilot assessment was pazopanib for the treatment of advanced renal cell carcinoma. Using the list of pharmaceuticals that received market authorization between 1 June 2010 and 1 February 2011, the topic was chosen based on the preferences of WP5/JA1 member organizations and stakeholders. The assessment was also a multi-HTA-organization effort, including fifty-four individuals from twenty-two EUnetHTA member organizations from sixteen European countries. The work was divided in eight domain teams. Subsequently, results of these domain teams were gathered in one pilot report. Feedback on the outcomes of the pilot was collected throughout the project and through structured surveys. Details about the pilot and the surveys are described in a different article in this journal (Reference Kleijnen, Pasternack and Rannanheimo9).
Phase 3
In phase 3 (February 2012 – March 2013), the experience from the pilot as well as discussions between WP5/JA1 organizations were used to make the second version (v2.0) of the RE assessment Model. The changes were implemented by the coordination team, after which the domain teams from the first version of the RE assessment Model functioned as reviewers. Subsequently, the RE assessment Model was subject to consultation by WP5/JA1 organizations and the WP5/JA1 Stakeholder Advisory Group, that consisted of a group of content related experts from various stakeholders (June/July 2012), and the public (October/November 2012). At each round of consultation, all comments were compiled into one document and each comment was addressed with a response whether the suggested change was implemented and a reason for rejection (if applicable) or explanation of how the comment was processed.
RESULTS
Based on the comments of the consultations and discussions between WP5/JA1 organizations, the coordination team produced the final version (v3.0) of the RE assessment Model which was published on EUnetHTA's Web site in March 2013. The final version of the RE assessment Model (v3.0) is limited to four domains with thirty-nine assessment elements (see Table 1). Table 2 lists the main changes that were made to the RE assessment Model and the number of comments received during the consultation. Details on the nature of the comments are included in Supplementary Table 2, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000609. Only those comments are included in Supplementary Table 2 for which there was consensus among the respondents.
Table 1. Assessment Elements of the HTA Core Model for Rapid Relative Effectiveness Assessment of Pharmaceuticals
AE, assessment elements
Table 2. Changes That Were Made during the Production of the HTA Core Model® for Rapid Relative Effectiveness Assessment and Results of Consultations
Phase 1
In the first version, the existing text of the HTA Core Model® was adapted for the purpose of undertaking a rapid assessment of pharmaceuticals and focusing more on the comparative nature of the assessment. In addition, the number of assessment elements was downsized. The most relevant comments from WP5/JA1 members on V1.0 included a preference for one general methods section instead of domain specific sections, the need for a specific “relative effectiveness” section and more emphasis on the comparative nature of the assessment.
Phase 2
The main points for improvement that were gathered from the pilot assessments included the following: an extensive literature search was not considered feasible (too time consuming), the general scope should include outcomes, overlap between assessment elements should be reduced, the number of assessment elements should be narrowed down and the last four domains were considered less relevant for a rapid assessment.
Phase 3
The most visible adaptations were implemented in the second version of the RE assessment Model as a result of the experience from the pilot and comments from WP5/JA1 members on version 1.0 that had not been addressed yet. First, the Social, Ethics, Organizational and Legal domain were excluded from the RE assessment Model as it was believed that these domains are out of scope for a RE assessment (which focuses on the clinical benefits and harms), they require too much work within the restricted timelines and they are more context specific and, therefore, less suitable for cross-border assessment. However, it was believed that for specific pharmaceuticals some social, ethical, organizational, and legal aspects may still be relevant, especially at the scoping phase of a RE assessment. Therefore, a checklist with questions was developed and included in the RE assessment Model (see Supplementary Table 1). Because the assessment is comparative in nature, only new issues should be dealt with, which arise from a difference between the pharmaceutical to be assessed and its major comparator(s). If a question in the checklist is answered with “yes”, further analysis of these issues may be warranted. Second, the project scoping was altered from Technology, Indication, Comparator (TIC) to Patient, Intervention, Comparator, Outcome (PICO). Whereas in the HTA Core Model® outcomes are determined per domain (not in the general scope), in the RE assessment Model outcomes are included in the general scope of the assessment. This is possible as there is less variance in outcomes as only domains with clinical outcomes are included. In addition, it was regarded preferable to have a central discussion on the relevant outcomes because it is crucial for correct scoping of a RE assessment. To reduce overlap in methodological guidance, one general methods section was developed for the RE assessment Model in contrast to the HTA Core Model® that contains methods sections per domain.
The traditional HTA Core Model® is based on the principle of doing a de novo assessment including an extensive literature search. This was changed for the RE assessment Model as this is not feasible in the short timeframe of a rapid assessment. The methods section of the RE assessment Model was based on the assumption that a submission file by the marketing authorization holder (including a systematic literature search) and European Public Assessment Reports are the primary sources for undertaking the assessment. The European Public Assessment Report is produced by the European Medicines Agency as part of the market authorization process for a pharmaceutical. There was a further exclusion of assessment elements that were not considered relevant. Finally, guidance was included in the RE assessment Model on how to produce a “relative effectiveness section” that combines data from all domains included (synthesis).
The main comments on v2.0 from the WP5/JA1 members, Stakeholder Advisory Group and public consultation are included in Supplementary Table 2. Most of these were addressed in the final version of the RE assessment Model. The main changes that led to the final version (v3.0) of the RE assessment Model were textual revisions, reduction of overlap between text in RE assessment Guidelines and RE assessment Model and further exclusion of assessment elements that were not considered relevant.
The following comments were not addressed in the final version of the RE assessment Model. First, a considerable number of comments from stakeholders, such as industry, related to the procedure of performing an assessment, including stakeholders involvement. It was decided that procedure related aspects should not be part of the RE assessment Model, as the focus of the Model is methodology and guidance for reporting, and that a separate procedure manual would be drafted for WP5/JA2 for doing cross-border assessments of pharmaceuticals. In addition, stakeholders requested to include more information on different types of modeling to extrapolate efficacy data to effectiveness data. It was decided to not include this information as these methods are still very much in development and not considered common practice by most HTA organizations. The request to exclude methods for post-authorization rapid assessments and solely focus on pre-authorization rapid assessments was rejected as WP5/JA1 organizations wanted the RE assessment Model also to be suitable for the reassessment of products for which new information is available. Furthermore, stakeholders pointed out that the RE assessment Model does not provide guidance for the marketing authorization holder on the required content of the submission file. Providing this guidance was considered out of scope for the current development of the RE assessment Model as the development of a template for a RE assessment submission file is planned for EUnetHTA JA2. Finally, the WP5/JA1 organizations agreed with stakeholders that information produced with the RE assessment Model should limit duplication of the content of regulatory documents. However, further testing of the RE assessment Model in pilots should show whether this requires adaptation in the RE assessment Model itself.
DISCUSSION
The aim of this article was to describe the development of the Model for Rapid RE assessment which allows standardized production and reporting of RE information. The current RE assessment Model was based on the HTA Core Model® and produced through collaboration of 18 European HTA organizations. The final RE assessment Model deviates from the traditional HTA Core Model® used for assessing other types of technologies. This is due to the limited scope (strong focus on RE), the timing of the assessment (just after market authorization), and strict timelines (e.g., 90 days) required for performing the assessment. The RE assessment Model was reduced to only the first four domains of the HTA Core Model® (instead of all nine) and a reduced number of assessment elements, thirty-nine research questions in total, in these domains. Furthermore, it was decided that the primary information sources should preferably be a submission file provided by the marketing authorization holder and the European Public Assessment Report.
The actual number of assessment elements included in an assessment could be lower than thirty-nine. If a specific assessment element is considered not relevant for the pharmaceutical(s) being assessed it should be excluded. Although the number of assessment elements was reduced substantially it is still a considerable amount of work to process them all in a rapid assessment. An argument for further downsizing the number of assessment elements is that avoiding overlap and duplication of information between assessment elements is more relevant for a rapid RE assessment compared with a traditional assessment produced by an HTA Core Model® as it is more likely that not only individual assessment elements will be used for national reporting but the whole assessment. The “report” is more sharable between countries due to the generic (not context specific) information in the RE assessment . Hence, the flow of information throughout the assessment becomes more relevant and the summary becomes a crucial part of the concise document instead of the detailed reporting of the results cards. Further piloting may show whether more of these assessment elements are abundant for a rapid RE assessment, enabling further downsizing of the RE assessment Model.
Whereas medicines regulatory agencies focus on the benefit-risk assessment of a pharmaceutical to establish whether the pharmaceutical will do more good than harm in a defined group of patients (Reference Eichler, Bloechl-Daum and Abadie1;Reference Henshall, Mardhani-Bayne, Frønsdal and Klemp10), the assessment of the RE focuses on the achieved health outcomes relative to comparative treatment options. In addition, in comparison to licensing assessments, RE assessments place more focus on clinically relevant outcomes (e.g., overall survival) and put more emphasis on the applicability (external validity) of data (Reference Eichler, Bloechl-Daum and Abadie1). The product of the licensing assessment (e.g., European Public Assessment Report in Europe) is considered a relevant source of efficacy and safety information for doing a RE assessment of pharmaceuticals (Reference Kleijnen, Goettsch and d’Andon4), however, due to the above mentioned differences, it is not considered suitable as direct input (as such) for reimbursement decision making.
One of the issues that is still under debate is the final assessment of the RE in which the clinical benefits and harms are adequately combined. The current RE assessment Model guides how to produce the relative effectiveness section (synthesis) in which information about the clinical benefits and harms should be presented in combination. However, combining the clinical benefits and harms in the RE assessment model only consists of visual parallel presentation of both data and not of one quantitative answer that combines both data. In the future, it would be preferable to achieve a quantitative way to illustrate the balance of benefits and harms. However, these methods are, as yet, not well developed or used. The European Medicines Agency and the Committee for Medicinal Products for Human Use have also not been able to identify established quantitative methods to balance risk and benefits for the market authorization process (11;12). If RE is combined with health economic assessments, which is current practice in assessments for the National Institute for Health and Care Excellence in the UK, it might be suggested to use quality-adjusted life years (QALY) as a combined outcome for the assessment of benefits and harms. But the use of the QALY is still under debate, especially outside economic assessment, and preferences and ideas on the use of QALYs deviate around Europe.
As discussed by Kleijnen et al. (Reference Kleijnen, Pasternack and Rannanheimo9), the pilot with the RE assessment Model and guidelines also showed that the traditional way of collaborating that has been used for the HTA Core Model®, in which many organizations are included as author of the assessment, does not seem viable for producing a rapid assessment. Instead, it is suggested that a few authoring agencies (e.g., two) per assessment would be preferred to avoid duplication of work and information, to increase consistency throughout the assessment and have more efficient communication between authors. The combination of the authoring agencies could vary according to needs and topics. Broad participation from various countries could still be ensured by involving several organizations in an in-depth review of the assessment.
The RE assessment Model was developed as a result of the recommendations of the HLPF on RE assessment. The HLPF emphasized the importance for EU Member States to exchange information on (the production of) RE assessments, but also acknowledged that decisions on the detailed operation of undertaking RE assessments, are most appropriately made at a national level (7;8). The RE assessment Model and Guidelines were developed to facilitate standardized production and sharing of RE information while also leaving room for divergence in methods and interpretation, if desired, between European Member States. For example, the RE assessment Model and Guidelines do not oblige the inclusion or exclusion of specific study types (e.g., observational studies) or the use of specific outcomes (e.g., surrogate outcomes) as there may be diverging views between Member States on the value of this type of data. However, the Guidelines help the assessor to make choices regarding methods and the RE assessment Model facilitates transparent reporting of the methods used. Other methodological principles on RE assessment defined by the HLPF have been respected as well such as addressing transparently uncertainty in the evidence base, addressing issues regarding translating evidence on relative efficacy into conclusions on relative effectiveness, comparison with the most appropriate healthcare interventions and identifying areas in which the evidence base on an intervention could most usefully be developed in the future.
The RE assessment Model is a European project, however, it could also be useful for countries outside Europe. So far, to the best of our knowledge, no standardized tools have been developed that aim at providing RE information that is useful beyond the national context. The increased interest in comparative effectiveness data worldwide and increased resources for these study types (e.g., through the American Recovery and Investment Act of 2009 [13]) indicates that outside Europe there is also a need for assessing the relative or comparative effectiveness of health technologies. It is, for example, expected that formulary decisions in the United States will increasingly be based on comparative effectiveness, whereas there is a need for guidance and training on how to use these data (Reference Biskupiak, Dunn and Holtorf14). As European countries have national/local legislations on coverage of pharmaceuticals, the RE assessment Model was developed to suit multiple countries. Therefore, it could also suit countries outside of Europe.
A limitation in the development of the RE assessment Model is that, due to time and resource constraints, the development process included only one pilot assessment (Reference Kleijnen, Pasternack and Rannanheimo9). It is uncertain whether the lessons learned from this pilot would also be applicable to other assessment topics. On the other hand, we believe that the experience involved in this project (multiple HTA organizations who perform assessment of various pharmaceuticals on a day-to-day basis) warrants that the adaptations made to the RE assessment Model are in general applicable to RE assessments. Further piloting of the RE assessment Model in the years to come, using a range of pharmaceuticals, should show the usefulness and provide input for further development.
In conclusion, the commitment of so many organizations and countries in the development of tools for RE assessment shows that there is a clear need for standardized RE information, ultimately resulting in assessments with better quality and less duplication of information. The RE assessment Model, including nine methodological Guidelines, was developed to facilitate standardized RE information. Possible future improvements can be made by refining the assessment elements and new methodological guidance on relevant areas.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2 http://dx.doi.org/10.1017/S0266462314000609
CONTACT INFORMATION
Sarah Kleijnen, MSc, in Biology, MSc in Health Economics Policy and Law and Project Coordinator EUNetHTA WP5 (skleijnen@zinl.nl), Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Pharmacoepidemiology and Clinical Pharmacology, Faculty of Science, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB, Utrecht The Netherlands
Iris Pasternack, MD, and HTA professional, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Summaryx Ltd. Untuvaisentie 2 A 19, FI-00820 Helsinki, Finland
Marc Van de Casteele, MD, PhD and co-ordinator experts pharmaceuticals, Rijksinstituut voor ziekte- en invaliditeitsverzekering, Directie Farmaceutisch Beleid, Tervurenlaan 211, B-1150, St. Pieters-Woluwe, Belgium and KU Leuven, Herestraat 49, 3000 Leuven
Bernardette Rossi, Bachelor of Pharmacy (Honours) and Masters of Science in Agriculture and Veterinary Pharmacy, Directorate for Pharmaceutical Affairs (DPA), Ministry for Energy and Health, Administration Building, St. Luke's Hospital, G' Mangia, Malta
Agnese Cangini, PhD, and Health economist, Italian Medicines Agency, Via del Tritone 181, 00187, Rome, Italy
Rossella Di Bidino, PhD, in Management of Healthcare Organizations, MSc in International Health Care Management Institutional, HTA Unit, “A.Gemelli” Teaching Hospital, Largo Gemelli 8, 00168 Roma, Italy
Marjetka Jelenc, MD, PhD and Senior Researcher, National institute of Public Health, Trubarjeva 2, 1000 Ljubljana, Slovenia
Payam Abrishami, MD, MA and Advisor, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands
Ilona Autti-Rämö, Research professor and chief of health research, The Social Insurance Institution, Research Department Nordenskiöldinkatu 12, 00250 Helsinki
Hans Seyfried, Master of Law, Hauptverband der österreichischen Sozialversicherungsträger, PO 600, Kundmanngasse 21, 1031 Wien, Österreich
Ingrid Wildbacher, Mag, PhD, Hauptverband der Österreichischen Sozialversicherungsträger, PO 600, Kundmanngasse 21, 1031 Wien, Österreich
Wim G. Goettsch, PhD, and Project Leader EUnetHTA WP5, Zorginstituut Nederland, PO Box 320, 1110 AH Diemen, The Netherlands & Pharmacoepidemiology and Clinical Pharmacology, Faculty of Science, Utrecht Institute for Pharmaceutical Sciences, P.O. Box 80082, 3508 TB, Utrecht The Netherlands
CONFLICTS OF INTEREST
Sarah Kleijnen, Iris Pasternack, Bernardette Rossi, Agnese Cangini, Payam Abrishami, Ilona Autti-Rämö, Hans Seyfried, and Wim Goettsch report grants to their institutions from the European Commission. Iris Pasternack also reports travel expenses covered by ISPOR. Ingrid Wilbacher reports grants from HVB (the association of the Austrian Social Security Organizations). The other authors report they have nothing to disclose.
