Health systems, mobile and digital health, and health information technology are enabling greater electronic capture of a wide range of health data; there are an increasing number of new therapies (e.g., cancer drugs) available by means of expedited regulatory pathways, requiring careful postmarketing surveillance, with limited or no evidence from randomized clinical trials (RCTs). There is also an increased interest in complementing RCTs with data from real-life settings (e.g., to better understand patient outcomes). In these contexts, the use of real-world data (RWD) is high on the agenda of regulatory agencies, health technology assessment (HTA) agencies, as well as other relevant stakeholders in the field of HTA (e.g., patient organizations, industry, clinicians, payers, and scientific societies).
For example, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force on RWD stated that “health decision-makers involved with coverage and payment policies are increasingly developing policies that seek information on ‘real-world’ outcomes” (Reference Garrison, Neuman and Erickson1). The topic of RWD and real-world evidence (RWE) (Table 1) (2) has also been highlighted during several Global (GPF) and Regional Policy Forums (RPF) of HTA international (HTAi), involving strategic-level discussions between HTA leaders from public and private sector organizations (Reference Husereau, Henshall and Jivraj3–Reference Mundy, Trowman and Kearney6).
Around the globe, RWE is used for several purposes particularly for product development (e.g., providing evidence on the natural history and epidemiology of a disease) and by regulators for postmarketing surveillance of safety issues (e.g., Sentinel for monitoring medical products that are regulated by the U.S. Food and Drug Administration [FDA]). It is also used by regulators for risk-benefit re-assessments, although on a more ad-hoc basis (7). In addition, both HTA agencies and regulators are beginning to consider the importance of RWE broadly in their deliberations on the value of health technologies (Reference Murphy, de Léséleuc and Kaunelis8–Reference Plueschke, McGettigan and Pacurariu11), it may provide information for outcomes-based managed entry agreements, and some authors have noted the potential for RWE to rationalize use of health technology and drive disinvestment decisions.
RWE is increasingly seen as providing a new form of knowledge that augments traditional clinical studies, that is, the evidence from different sources is considered. However, some key challenges are apparent in the production and use of RWD to ensure that it can yield reliable RWE for policy relevant questions in the area of HTA. This is, for example, described by several authors as compiled in Makady's 2018 PhD thesis (Reference Makady12). In addition, Bowrin et al. (2019) found that, although most HTA guidelines describe the benefits and limitations of using RWE, there is little formal guidance on how to use RWE in modeling the cost-effectiveness of health interventions (Reference Bowrin, Briere and Levy13). Other authors have also acknowledged that the tide of RWD is approaching and that it is now time for action (14). Therefore, the central theme of the 2019 HTAi GPF focused on defining actions for the HTA community to address the key challenges and to properly develop and manage RWD /RWE to inform decision making across the technology lifecycle.
Specific key challenges related to the theme were identified during the June 2018 HTAi GPF scoping meeting, and include: (i) quality and acceptability, (ii) governance and accountability, (iii) transferability, and (iv) informing decision making.
The key challenges, as well as different initiatives in the field of RWD/RWE, were described in a background paper that served as an input to the 2019 HTAi GPF (Reference Oortwijn15).
In this article, we provide a reflection of the discussions of the above-mentioned challenges and the opportunities for addressing these that occurred during the 2019 HTAi GPF. This article is not a consensus statement of the attendees. As such, it cannot be taken to represent the views of any of the individuals attending the meeting or of the organization for whom they work.
Methods
During the 2019 HTAi GPF, held on January 27–29, seventy-three representatives from public sector HTA agencies, industry, HTAi leadership and invited distinguished speakers (Supplementary Table 1) met under the Chatham House Rule (16).
The 2019 HTAi GPF was structured with keynote presentations from three invited experts who shared their experiences of producing and using RWD/RWE for health technology decision making (i.e., producer and user of RWD/RWE, HTA/payer, and a patient representative). Thereafter, case studies were presented by seven GPF members to show how they currently address some of the identified key challenges for developing and using RWD/RWE to inform decision making. Visions for the future of HTA, taking into account the evolving landscape of generating and using RWD/RWE, were presented by four GPF members, reflecting the perspective of a HTA agency, industry, provider/payer and patients. All the presentations were used as a basis for discussions in several groups and during plenary discussions.
To encourage interactive group discussions, Design Thinking methodology was used. Design Thinking is useful in tackling complex problems, and has been applied in many sectors. The method consists of an iterative design process, including several steps to be followed to focus discussions. Several frameworks exists for taking these steps. During the GPF, we used the framework of Liedtka and Ogilvie of the Interaction Design Foundation (17), asking four specific questions: (i) What is? Exploring the current problem; (ii) What if? Envisioning alternative futures; (iii) What wows? Getting attendants to make choices about the identified futures; and (iv) What works? Making it work in practice.
Before the start of the GPF, group facilitators were assigned and instructed by a professional teacher about the framework and steps to be taken.
Results
The GPF addressed each of the key challenges to properly develop and manage RWD and RWE in the field of HTA. Below, the main discussion points are described, as well as what the HTA community can contribute regarding using RWE in the context of HTA processes. The main actions that were suggested by the GPF to be taken forward by the HTA community are listed in Table 2.
Table 2. Actions Suggested Advancing HTA in the Area of RWD/RWE
Quality and Acceptability
The following question was discussed in detail: how should the quality of RWD/RWE be guaranteed, so that it can be used for HTA purposes and be acceptable to decision makers? Attendees of the 2019 HTAi GPF emphasized that the most important issue is to have a common understanding of what type of HTA questions RWE is appropriate to answer. In the discussion, the challenge of addressing bias (e.g., under- or overestimating true effects) (Reference Karim and Booth18) was highlighted by the attendees. Specifically, the importance of describing and understanding the level of uncertainty in RWE that decision makers, including regulators, are willing to accept was emphasized. The issue of when and how an RWD analysis can assess causality was another area of debate (Reference Gerstein, McMurray and Holman19). In terms of opportunities, it was emphasized that the collection and analysis of RWD needs to be robust and transparent. Reliable analyses are required to generate trustworthy results. It was, therefore, recognized that significant effort is required to clean RWD, ensuring quality control and reproducibility of the dataset(s) by agreeing on common data models and developing quality standards for RWD/RWE. Where possible existing standards or guidance that are applied could be used (e.g., on coding of clinical data) (14), and EUnetHTA Joint Action 3 (JA3) is developing a tool to determine the quality of RWE registries for HTA purposes (Reference Oortwijn15). Also, the interoperability, and replicability across different datasets were mentioned as critical prerequisites.
Trust and transparency in the process of generating and analyzing RWD were also highlighted and indicated to be the responsibility of all relevant stakeholders. With regard to increasing transparency, we refer to the report of the ISPOR and the International Society for Pharmacoepidemiology (ISPE) Special Task Force on RWE in Health Care Decision Making (Reference Berger, Sox and Wilke20). This report (2017) describes good procedural practices that would enhance decision makers' confidence in using RWE. Finally, it was noted that HTA agencies do not have the skills in their workforce (in particular, data scientists) to be able to advise on RWE studies or to critically appraise them. Increased collaboration with those that are capturing (e.g., Google, Clinical Trials Transformation Initiative, public hospitals) and analyzing RWD (e.g., Flatiron, Patient-Centered Clinical Research Network [PCORnet], academics), including those outside the health sector (e.g., those working in the field of machine learning, artificial intelligence, or data science) is a necessary step for the immediate future.
Governance and Accountability
RWD is being produced by different stakeholders and for different purposes. In most cases, there is a lack of incentives for data sharing; limited or nonexistent standards for collaboration between stakeholders; while patient consent, privacy, and data security are recurrent themes of concern. All these challenges are hampering data access and can result in high costs for data protection to comply with relevant regulation (e.g., data privacy laws) (Reference Oortwijn15). During the 2019 HTAi GPF, attendees discussed how to improve the governance and accountability of RWD for producing RWE to inform HTA. Good practice guidance on multi-stakeholder data ownership and management was stated as important, as well as defining what data can be shared with and between HTA agencies and regulators.
For the longer-term, the establishment of a global directory of accredited data sources, potentially funded by a transparently operated global public/private partnership and an ethical framework for collecting and using RWD were mentioned as laudable aims. Any sustainable governance structure for generating and accessing RWD should respect data privacy obligations and involve all stakeholders, as recommended by Cole et al. (Reference Cole, Garrison and Mestre-Ferrandiz21). For the enhancement of transferability (discussed below), the development of an international or cross-country governance framework could be a way forward. Such a framework should take into account the governance issues mentioned above, as well as methods for aligning different data sources to capture core outcomes, open data, and standards to ensure transferability from RCTs to RWD/RWE where applicable.
Transferability
Differences in structure, setup and content of databases can lead to challenges in sharing data and information across countries and/or regions. The issues of what, when and how to transfer RWD and RWE across jurisdictions were discussed. For example, conducting multi-country pilots with well-defined patient groups and interactions (e.g., in rare diseases) was listed as an option. HTA agencies involved in EUnetHTA JA 3 are conducting such pilots in which they agree on RWD to be collected for a specific health technology or disease. It was also emphasized that the patient-clinician relationship should be at the heart of RWD collection to support shared decision making at the patient level, with the data then being used at an aggregated level to improve health services. It was noted that bringing different RWD collection communities together could be a way forward; that is, sharing experiences on what is working well and what can be implemented efficiently to generate qualified and meaningful RWD.
For example, patient care management can be supported by technical platforms that bring together systems and processes to support optimal care for the individual patient and aggregate that data to develop a more efficient health system. For example, the Chordoma Foundation collaborates with the Children's Hospital of Philadelphia in the United States, aiming to accelerate and improve bio-specimen based research in laboratories around the globe. For this purpose, they make use of “an innovative healthcare discovery ecosystem that creates open-access platforms which integrate complex genomic and clinical patient data” (22).
In addition, replication of studies was mentioned as an important requirement for transferability. In this context, research questions (defined in terms of Populations, Interventions, Comparators, Outcomes, Timing, Setting: PICOTS) could be shared across jurisdictions regarding the same pathologies and interventions. Furthermore, agreement on the definition and prioritization of patient-focused endpoints to allow replication and cross-validation would be helpful. To enable this, engagement with both clinicians and patients is needed. It was also mentioned that better (ICT) practices and incentives for clinicians would support getting better data; that is, fitting the data collection within their daily workflow to lessen the burden of administration. Finally, the need to promote data collection in clinical settings was emphasized. A best practice example in this context is PCORnet. PCORnet aims to conduct patient-centered outcomes research more efficiently and faster by leveraging electronic health record data and administrative claims data, and partnerships with relevant stakeholders (23).
Informing Decision Making
Currently there is no clear and widely accepted guidance on when and how to use RWD/RWE to inform decision making. It was extensively discussed when it would be appropriate to use RWE, for example related to providing scientific advice (early interactions with HTA) and for informing decisions such as coverage, outcome-based contracting, and disinvestment. In addition, the necessary conditions for using RWD/RWE across the technology life cycle were debated. For some time, there have been discussions about when, how, and why RWE should be used in the context of HTA (Reference Oortwijn15). A recent example of effective collaboration between relevant key stakeholders in the development and use of RWE for coverage and formulary decisions of pharmaceuticals was presented, based on the work of Pearson et al. (2018).
The authors describe the development of a common framework, including the necessary evidence standards for the question that it is intended to support, and the context in which the decision needs to be made (Reference Pearson, Dreitlein and Towse24). The 2019 GPF attendees believed that meaningful stakeholder engagement, initiated by HTA agencies (information brokering), is important, especially because RWD is increasingly available and industry (e.g., Sanofi's DARWIN platform) and others (e.g., NHS Digital in the United Kingdom) are already building their own datasets and platforms, and have significant capability to analyze RWD. However, the current HTA systems are not fully equipped to address these developments in terms of capacity and evaluation methods (e.g., data science). It was, therefore, highlighted that the HTA community is currently standing at a cross-road and that action is urgently needed.
Policy Implications – from Theory to Action
In moving forward, all the stakeholders in the HTA community need to align on framing the relevant (research) problem and on when to generate and use RWD/RWE across the technology lifecycle. The community should also start aligning on what the best sources of evidence are according to the research problem and how the data should be collected to create evidence that can be relied upon. The community is urged to work more closely together, and include groups outside the traditional HTA community and health sector to build capacity in the area of big data analysis. Furthermore, the HTA community should instruct technology developers (e.g., as the FDA already does with the help of Flatiron) and become influential as to what data are needed for answering which questions to informing decisions (e.g., developing a routine pathway for use of RWD in early dialogue or using RWD/RWE to advance dynamic pricing models).
To support this movement, it is necessary to have more clarity on methods, standards and data sharing, more streamlining in data collection (common data models), and integration of RWD collection in routine health care practice. Therefore, databases for use across jurisdictions, detailing the methods, sources, and replication of RWD, using advanced rigorous and sound methods and expertise, as well as building trust are important features to collaborate on.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462319000400.
Conflicts of interest
All authors received funding from HTAi for the work reported in this study.
