The use of registries is becoming increasingly common in health technology assessment (HTA) as interest grows in the use of observational data to complement experimental data and to accelerate the process of access to new technologies (1). Registries have been defined as “an organized system that collects, analyses, and disseminates the data and information on a group of people defined by a particular disease, condition, exposure, or health-related service, and that serves a predetermined scientific, clinical or/and public health (policy) purposes” (Reference Zaletel and Kralj2). The quality of registry data has often been criticized, however, leading to reluctance to embed their use in HTA (Reference Dhruva, Bero and Redberg3–Reference Smeeth, Douglas and Hubbard5). While there are several guides to improving observational data collection and reporting, there is no standardized tool for use by HTA agencies to assess registry quality (Reference Glynn, Campbell, Marlow and Patrick6–8).
The European network for Health Technology Assessment (EUnetHTA) has been working through a series of work packages of its Joint Action 3 (2016–19; referred to as EUnetHTA JA3) to enhance the use of high-quality registries in HTA. The purpose of one of these work packages (Work Package 5 Strand B) is the production of a standardized tool for the use of registries in HTA, based on the recommendations of the “Methodological guidance on the efficient and rational governance of registries” (referred to here as the PARENT Guidelines) (Reference Zaletel and Kralj2). The PARENT guidelines describe important dimensions in assessing the quality of registries, including governance, data quality, information quality, and data protection (Table 1). The aim of the guidelines was to support EU Member States in developing comparable and interoperable patient registries in fields of identified importance (e.g., chronic and rare diseases, medical technology) with the aim to rationalize the development and governance of patient registries, thus enabling analyses of secondary data for public health, policy and research purposes in cross-border settings.
Table 1. Recommendations for Registries from the PARENT Guidelines
We present here the findings from the first part of this work package, namely (i) a literature review to identify any existing guidelines and/or assessment tools for quality assurance of registries, (ii) a survey to explore the current understanding and use of registries by HTA agencies and particularly the employment of any standards/criteria or other tool to assess the quality and comparability of registries before their use in HTA, and (iii) an overview of the registry quality dimensions in the standards/criteria identified through the literature review, researchers’ prior knowledge, and the survey. The purpose of the literature review, the survey and the overview of the guidance documents for registries was to feed into the development of a standardized tool to assess registry quality.
METHODS
The National Institute for Health and Care Excellence (NICE) in the United Kingdom (UK) and the Croatian Institute of Public Health (HZJZ) led this study on behalf of EUnetHTA JA3. We conducted a literature search, using PubMed as the literature database and the following search terms were used “(“technology assessment, biomedical”[MeSH Terms] OR (“technology”[All Fields] AND “assessment”[All Fields] AND “biomedical”[All Fields]) OR “biomedical technology assessment”[All Fields] OR (“technology”[All Fields] AND “assessment”[All Fields] AND “biomedical”[All Fields]) OR “technology assessment, biomedical”[All Fields]) AND (“registries”[MeSH Terms] OR “registries”[All Fields])”. There were no date restrictions but only articles in English were reviewed. A single reviewer examined titles and abstracts to identify those that referred to the use of registries for health technology assessment.
An initial survey tool was developed based on discussions between NICE and HZJZ on the work program objectives. Dimensions of registry use to be assessed in the survey included: the use of different types of registries by HTA organizations, the purposes for which registries were used in HTA, and the methodology and processes applied to assess the quality of registries before use in HTA.
Types of registries to be included in the survey were based on the PARENT Guidelines and defined as follows: Disease/condition registries (include patients with a common disease or condition, for example, cystic fibrosis or cancer); Pharmaceutical registries (include patients who have taken a particular pharmaceutical product); Medical technology registries (include patients who have been exposed to a particular device or diagnostic technology); Procedural registries (include patients who have undergone a particular medical or surgical procedure).
We specified purposes for which registry data could be used by HTA agencies to reflect steps in the HTA process, namely: Natural history of disease/condition; Evaluation of effectiveness (for example, data on the natural history of a disease/condition for decision modelling, or to create cohorts for comparative effectiveness analysis); Evaluation of cost and/or budget impact (for example, cost data from pharmaceutical registries, current and/or potential uptake of health technology from disease/condition registries); Future reviews of the technology, particularly where there is a lack of evidence for the technology (for example, safety/adverse events data from medical technology or procedural registries).
We asked whether HTA agencies used any standards or criteria to assess the quality of registries before use, and if so, whether these were defined internally by the organization or an external organization. If no standards were used, we asked whether other steps were taken to evaluate quality of registry data before use in HTA.
The survey was conducted in English. Pretesting of the survey tool was conducted among EUnetHTA JA3 members of NICE and HZJZ, with adjustments made to the definitions of types of registries and the addition of a hyperlink to the PARENT wikipage on quality. A formal pilot of the survey was then conducted with two HTA organizations selected to ensure European and non-European representation, namely A Unidade de Asesoramento Científico-técnico (Avalia-t) in Spain and the Health Intervention and Technology Assessment Program (HITAP) in Thailand. Changes based on feedback from these organizations included the addition of a question on other methods to assess the quality of registries apart from quality standards, and revision of some wording to improve clarity for nonnative English speakers. A final survey tool was developed to reflect these changes (see Supplementary Table 1 in the online material).
The final survey tool was sent by email to all EUnetHTA JA3 partner organizations and all HTA organizations in the International Society for Pharmaco-economics and Outcomes 7 Research (ISPOR) directory; a total of 110 organizations (9). One reminder email was sent after 2 weeks to all organizations that had not yet responded. The survey was closed 1 month after the initial call.
Participants who reported using internal standards to assess the quality of registries were contacted up to three more times to request that they provide a copy of those standards for review. We compared the criteria listed in the external and internal standards obtained, with the recommendations of the PARENT project.
RESULTS
The literature review returned 96 titles and abstracts that met the inclusion criteria, from which we identified 22 relevant publications. The review identified no standards or guidelines specifically relating to the use of registries for HTA; however, several described attributes of high quality registries, which we discuss further below.
We received 55 responses to the survey from organizations representing 21 different countries, a response rate of 40.5 percent (43/110). One organization was excluded as it does not undertake HTA (Semmelweis University Health Services Management Training Centre in Hungary). Two responses were received from 6 organizations: the Canadian Agency for Drugs and Technologies in Health (CADTH), Finnish Medicines Agency (FIMEA), Association of Austrian Social Insurance Institutions (HVB), Scottish Medicines Consortium, Belgian Health Care Knowledge Centre (KCE) and Swedish Dental and Pharmaceutical Benefits Agency (TLV). For these organizations, we used the first response received for analysis.
Responses were received from across Europe (see Supplementary Figure 1). Agencies in Canada and Thailand also provided input. Responses were received from 33 of a total of 78 (42 percent) EUnetHTA partners. No responses were received from HTA organizations based in Latin America, Africa, or Australasia.
Disease/condition registries were the most common type of registry used in HTA, with nearly three quarters of responding organizations using these registries compared with half or less using pharmaceutical, medical technology, or procedural registries (Table 2). Other types of registries used in HTA included health expenditure databases such as reimbursement or insurance data, pharmaceutical or medical technology wholesale data; clinical trials registries; and routine databases for usual care.
Table 2. Types of Registries Used in HTA
HTA, heath technology assessment.
Effectiveness data and estimation of the current and/or potential uptake of a health technology were the two most common uses of registry data, with over two-thirds of responding organizations using registry data for these purposes (Table 3). Nearly two-thirds of organizations were using registry data to estimate safety or adverse events. Registries were also being used by around one in two organizations to provide data on costs, the natural history of a disease or condition and cohorts for comparative effectiveness analysis. Other uses included assessment of comorbidities and patient characteristics for managed entry agreements and to monitor the appropriate use of pharmaceuticals postlaunch.
Table 3. Use of Registry data in HTA
HTA, health technology assessment.
Sixteen organizations reported that they use criteria or standards to assess the quality of registry data before use in HTA. Most organizations used internally defined criteria or standards (14/16; 87.5 percent) with one organization using both internally defined and external criteria/standards. Of the 14 organizations that reported using internally defined criteria, only two made these available to the study group (Italian Arthroplasty Registry and NICE). The Italian Arthroplasty Registry was excluded from further analysis as this was a review of data in the registry, rather than criteria for assessment of data quality. None of the organizations (other than NICE) using internally defined registry standards or criteria have published any assessments of registries using these tools.
For those organizations not using criteria or standards to assess the quality of registry data, nearly one in two used discussion with experts (13/27; 48.1 percent) and one in three used discussion with stakeholders (9/27; 33.3 percent). One in five (6/27; 22.2 percent) inspected registry data directly before use in HTA. One organization noted what information was lacking in existing registry studies. Another highlighted that there were no specific quality standards available for registries.
The literature review, researchers’ prior knowledge, and the survey identified 13 guidance documents for registries (Reference Zaletel and Kralj2,7,8, 10–19): Methodological guidelines and recommendations for efficient and rational governance of patient registries (PARENT) (Reference Zaletel and Kralj2); Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices (FDA) (7); Recommendations for the development and operation of health-related registries (ANQ) (8); Medical Device Registries, Six Key Principles (EUCOMED) (10); Evaluating databases (ReBIP) (11); Principles of International System of Registries Linked to Other Data Sources and Tools (IMDRF) (12); Interventional Procedures Programme manual (NICE) (13) Registries for evaluating patient outcomes: A user's guide (AHRQ) (Reference Gliklich and Dreyer14); Operating Principles and Technical Standards for Australian Clinical Quality Registries (ACSQHC) (15); Data Quality, Validation, and Data Source Integration in Rare Disease Registries (EPIRARE) (Reference Posada, del Otero and Villaverde16); The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies (Reference von Elm, Altman and Egger17); A Validated Checklist for Evaluating the Quality of Observational Cohort Studies for Decision-Making Support (GRACE Initiative) (Reference Dreyer, Velentgas and Westrich18); Registry Studies: Why and How (19).
These publications differed in their scope (real world data sources versus patient registries in general versus specific type of patient registry), purpose (conducting versus reporting versus evaluating research), dimensions covered (design and conduct versus quality dimensions such as governance, data quality, and safety), or format (checklist versus explanatory form). Three of the 13 guidance documents were excluded from further analysis because they focused on retrospective evaluation of design and conduct of a registry rather than prospective quality dimensions (Reference von Elm, Altman and Egger17;Reference Dreyer, Velentgas and Westrich18) or were based (19) on another guidance document (Reference Gliklich and Dreyer14). Table 4 provides a comparison of the included 9 quality guidance documents against the dimensions described in the PARENT guidelines.
Table 4. Comparison of PARENT Recommendations with Nine Relevant Published Guidance Documents
ACSQHC, Australian Safety and Quality Goals for Health Care; AHRQ, Agency for Healthcare Research and Quality; EPIRARE, European platform for rare disease registries; EU, European Union; Eucomed, the European trade association in the field of medical devices; FDA, United States Food and Drug Administration; IMDRF, International Medical Device Regulators Forum; N/A, Not applicable; NICE, National Institute for Health and Care Excellence; ReBIP, Review Body for Interventional Procedures.
For clarity, we have not provided the reference number of each corresponding criterion for each guidance, which are instead available in the Supplementary Materials online (along with summary details of each of the guidance documents in Supplementary Tables 1–13). Comparison is made against the PARENT guidelines, as these served as a starting point for the development of a standardized registry quality assessment tool, for which this work acted as additional input. Table 4 shows wide variation in the criteria covered across all standards. Only the AHRQ standards covered all the quality dimensions outlined in the PARENT guidelines, with the internal NICE standards showing the fewest corresponding criteria.
The most commonly mentioned areas across the guidelines are: Procedures and methods for registry operation and governance, Self-assessment, Data accuracy and completeness, Mode of data collection and impact on data quality, and Legal and ethical issues. With such consistency across guidelines, these areas could be viewed as essential quality criteria for education and training. Resource planning, Interpretable and accessible data, and Information quality (in terms of data briefings or recent publications) were omitted the most frequently and could be viewed as optional quality criteria. Interoperability, the key element to PARENT endeavors, was covered or explained in six of the nine guidelines, and it is mostly presented as semantic or technical interoperability, rather than being described through all five interconnected levels as it stands in the European Interoperability Framework and PARENT guidelines.
DISCUSSION
This survey of HTA organizations shows that many agencies, particularly in Europe, are drawing on a range of registries to provide data for their HTA. Less than half, however, currently use criteria or standards to assess the quality of registry data before use in HTA. Nearly all criteria or standards that are being used by HTA organizations have been defined by their organization, rather than a standardized tool published by an external body. A comparison of internal and external standards identified wide variation in content. However, there was consistency in several quality criteria, which can be used as a starting point for development of a standardized tool.
Registries have been recognized as an important source of data and information, both during the prelaunch as well as postlaunch phases of the technology lifecycle and related assessments. Yet until now, only anecdotal evidence was available on the use of registries by HTA organizations in Europe (10;11). Contrary to expectations, this survey shows that HTA organizations are actively using registry data for complex decision making in a range of areas but without reference to a standardized method to assess relevance and quality. Given that the use of registries is more prevalent than expected, this strengthens the need for a standardized tool to promote best practice for the collection and use of such data.
The survey also identified several criteria/standards currently in use by HTA organizations. We were only able to obtain one example of internal standards in use, despite concerted follow-up. It is possible that the internal “definition” of standards was not formalized and in an easily sharable form, which again supports the need for an accessible and rigorous tool. The challenge is to apply such standards consistently to ensure that only registry data of sufficient relevance and quality influences decision-making. A previous audit of registries using the NICE internal standards found that the quality of recommended registries was disappointing, with only a few registries mature enough to deliver evidence of sufficiently high quality to inform funding decisions (Reference Mandeville, Patrick, McKenna and Harris20). The NICE internal standards were found to be more limited than external guidelines in terms of quality criteria, which may have contributed to this result.
Our comparison of internal and external standards provides a good starting point for the development of an internationally recognized, user-friendly tool that can be used across jurisdictions. Such a tool developed in conjunction with EUnetHTA stakeholders will support consistency of application, as well as reassure critics who have traditionally considered registry data to be unreliable for use in HTA. Collaboration in development of such a tool will be essential to achieve agreement around the application of terminology. For instance “completeness” is considered by many to be a criterion that needs to be evaluated in the context of a registry purpose, recognizing that a registry may attempt to collect broad data to meet the interests of all stakeholders but not all may be essential to the purpose of HTA.
The initiative must also recognize that data quality assessment and management for evidence generation is highly topical currently and should learn from other relevant work, for example, “Data Curation” covers many of the principles that the tool should include. It has been defined as “the active and ongoing management of data through its life cycle of interest and usefulness to scholarship, science, and education. Data curation activities enable data discovery and retrieval, maintain its quality, add value, and provide for reuse over time, and this new field includes authentication, archiving, management, preservation, retrieval, and representation” (21).
Strengths of this research include the comprehensive piloting of the survey tool to ensure common interpretation among nonnative English speakers. Despite this, it is possible that the survey was subject to variation and overlap in use of terms such as disease/pharmaceutical registry. We included purposes of registry data specific to the needs of HTA agencies, that is, organizational (uptake), technological (effectiveness), or economical (cost) evaluation clusters; however, it is possible that registries were being used for other purposes not picked up as relevant to this survey, for example, epidemiological. Any standard developed will need to reflect the diverse current use of registry data in HTA.
The comprehensive distribution list used for the survey strengthened the methodology, but may have contributed to a fairly low response rate of 40 percent. Translation of the survey tool into other languages such as Spanish may have increased participation, for example from Latin American HTA agencies. Thus, the results presented here should be seen as only an indicative picture of the relationship between HTA activity and registries. An alternative methodological approach, rather than identifying quality standards already in use, would have been to build consensus on those registries that are considered to produce high quality data and then to examine features that these registries had in common. However, many registries only capture data from one jurisdiction and, therefore, it seemed likely that these features would be already captured in jurisdiction-specific quality standards.
In conclusion, many HTA agencies are already using registry data, despite the lack of a standardized quality assessment tool. A review of existing standards found wide variation in content, but some consistency in included and omitted criteria. These findings will be taken into consideration during the development of the EUnetHTA registries for HTA tool.
SUPPLEMENTARY MATERIAL
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462318000478
Supplementary Figure 1: https://doi.org/10.1017/S0266462318000478
Supplementary Table 1: https://doi.org/10.1017/S0266462318000478
Supplementary Table 2: https://doi.org/10.1017/S0266462318000478
Supplementary Table 3: https://doi.org/10.1017/S0266462318000478
Supplementary Table 4: https://doi.org/10.1017/S0266462318000478
Supplementary Table 5: https://doi.org/10.1017/S0266462318000478
Supplementary Table 6: https://doi.org/10.1017/S0266462318000478
Supplementary Table 7: https://doi.org/10.1017/S0266462318000478
Supplementary Table 8: https://doi.org/10.1017/S0266462318000478
Supplementary Table 9: https://doi.org/10.1017/S0266462318000478
Supplementary Table 10: https://doi.org/10.1017/S0266462318000478
Supplementary Table 11: https://doi.org/10.1017/S0266462318000478
Supplementary Table 12: https://doi.org/10.1017/S0266462318000478
Supplementary Table 13: https://doi.org/10.1017/S0266462318000478
CONFLICTS OF INTEREST
There are no conflicts of interest to declare.
