Many HTA agencies—not only in Europe but also worldwide—are working on similar topics at the same time. This insight is not novel and is mainly based on the HTA database for published reports (1). For this reason, the EUnetHTA project 2006–2008 (WP7) aimed to contribute towards avoiding duplication by making the information gathered on “new and emerging” technologies available to a wider audience beyond regional or national decision makers. The EUnetHTA Joint Action 1 2010–2012 (WP7) was built partly on the work achieved in the first phase of EUnetHTA. But then the focus was clearly on “new technologies” (only), defined as “after market approval, but before general or broad reimbursement.” The intention was to facilitate information flow and access to assessments of those new technologies before completion or publication. WP7 aimed at collecting information on ongoing and planned projects and assessments of new pharmaceutical and nonpharmaceutical technologies, at consolidating and synthesizing the information electronically in a web-based database equipped with searching and sorting options. Finally, it aimed at alerting those partners working on identical or similar projects and, therefore, easing collaboration among them to reduce duplication of HTA efforts.
This intention to reduce duplication and to ease collaboration has been a topic among HTA agencies for many years (Reference Kristensen, Lampe and Chase2–Reference Neikter, Rehnqvist, Rosén and Dahlgren4): Setting up such a database was a topic within the International Network of Agencies for Health Technology Assessment (INAHTA) (5;Reference Hailey6), but did not work out well due to the INAHTA members’ lack in actually sharing their information on ongoing projects in a timely manner so that others were informed before the publication of assessments.
While the established and widely used HTA Database at the Center for Reviews and Dissemination (CRD) (1), funded by the UK National Institute for Health Research (NIHR) in York hosts summaries of over 12,000 published HTAs and 20,000 quality-assessed systematic reviews, PROSPERO ((7) developed in 2011 and also hosted and managed by CRD) is a database of prospectively registered systematic reviews in health and social care, and contains (as of June 2013) 1,704 records of ongoing systematic reviews, including Cochrane protocols (Reference Booth8). Key features from the review protocol are recorded and maintained as a permanent record in PROSPERO to provide a comprehensive listing of systematic reviews registered at inception and enable a comparison of reported review findings with those planned in the protocol (Reference Booth, Clarke, Ghersi, Moher, Petticrew and Stewart9–Reference Booth, Clarke and Ghersi12). PROSPERO is also funded by the UK NIHR with the aim of reducing the commissioning of research that is already under way.
All of the three efforts to set up a database for ongoing and planned projects, whether they be Cochrane reviews and other systematic reviews or health technology assessments with a focus on reimbursement or coverage decisions, are based on the realization that there is a certain percentage of overlap in the international production of evidence syntheses. All three efforts are driven by the intention to reduce duplication and release precious public resources for an increased output of different, rather than overlapping, research results.
This article intends to present the development, structure and results of a database on planned and ongoing HTA projects (POP Database or POP DB) in Europe developed in EUnetHTA JA 1 and maintained in JA 2.
METHODS
A retrospective description of the POP DB design and its development from 2010 to 2013 was carried out by using EUnetHTA work and communication protocols and the annual technical reports. The description was followed by a detailed qualitative and quantitative analysis of the information and data.
Analytical categories are
-
• Database development: structure and functionalities;
-
• Procedures: database maintenance – updating and alerting, roles and statistics;
-
• Access: rules and management;
-
• Additional materials: work sheets, user manual, communication statistics;
-
• Content: POP DB statistics using the descriptive statistical analysis.
The information is presented and explained in a step-by-step manner; the numerical data are presented as frequencies and percentages.
RESULTS
The POP Database (POP DB) was set up in an iterative process from a basic Excel sheet to a multifunctional electronic online database. The functionalities, such as the search terminology, the procedures to fill and update the DB, the access rules to enter the DB, as well as the maintenance roles, were defined in a multistep participatory feedback loop with EUnetHTA Partners.
Database Development, Structure, and Functionalities
The first step to set up a database was taken in 2010 with a simple Excel sheet containing one sheet per agency (n = 56, all of them were Associated or Collaborative Partners within EUnetHTA) and one overview sheet including all projects. The partner sheets comprised information in English about the agency, its abbreviation and country, the main contact person and e-mail address, the titles of the projects, the status (planned, defined as project not started yet or ongoing, defined as project in the early beginning) and—if available—a website address for a detailed project description. In early 2010 the partners already decided that the titles alone would not fulfill the needs for searching through the list; therefore, an indexing and categorization system was needed. It was agreed to further assign at least two different Medical Subject Headings (MeSH) (http://www.nlm.nih.gov/mesh/) terms for classifying each project according to the type of technology used and the type of disease (a minimum of one and a maximum of three per category). The MeSH categorization was first piloted and then introduced. The actual assignment of MeSH terms for all (1,200) projects and the “clustering” of identical or similar projects was carried out (for almost 2 years) by a single researcher within the Ludwig Boltzmann Institute of Health Technology Assessment (LBI-HTA).
The development of the online POP DB started in late 2010 in cooperation with WP6 (Information Management) and was based on a survey (January 2011, fifty-seven questions, response rate 81 percent) on the requirements and needs perceived by the users and information providers of the former POP DB (Excel sheet). Interoperability with existing databases (HTA Database at CRD) was defined as an additional requirement. POP DB v1 (version 1, see Supplementary Figure 1: Screenshot EUnetHTA POP Database, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000567) was released in August 2011 and started with the same dataset, but also included more user-friendly functions such as
-
• Identify records (search: by keyword in title and description and by metadata [agency, country, entry date, status]; browse: by metadata and by POP MeSH browser; list potential collaborations [when at least two MeSH terms are in common: all similar projects to the reader's agency projects]).
-
• Result-presentation (output of table and results list: sortable by column headers).
-
• Notification (to moderator of POP DB by means of e-mail when a new record is created/published).
-
• Definition of workflow (see procedures for data maintenance).
Procedures: Database Maintenance – Updating and Alerting, Roles and Statistics
Reminders and requests to update one's own POP DB entries are sent out quarterly by LBI-HTA. While the Excel POP DB had two roles (creator and moderator), the POP DB v1 works with three roles in a predefined workflow: After having been reminded to update POP, the “creator” (usually one or two persons per agency) creates and publishes a record (enter data of new projects). This role allows the editor to edit (change status, etc.) the records of his/her own agency. The “moderator” role is to verify selected MeSH terms by the creator before they become visible. In contrast to the time-consuming MeSH assignments by LBI-HTA in the 1st phase, this role was moved to the creators. “Readers” can only read the content of the database and use the search, browse and show-potential-collaborations functionalities. Finally, the system automatically “unpublishes” records three months after publication (status: “published”) or “stopped” date (invisible).
After each round of quarterly reminders and according to the updates to the POP DB, the LBI-HTA then manually synthesizes the information into a list of identical and similar projects. Following this work step, the information is sent back to all EUnetHTA Partners by e-mail. Identical topics (so-called “alert topics”) are defined as those where the pathology (indication, patient group) and the technology MesH terms are identical. Similar topics are those where at least two MeSH terms (one per pathology and one per technology) match. Because no individualized information for each agency is sent out, searching for potential collaborations is left to be done by the agencies. Additional descriptive statistics—so-called communication protocols (number of agencies answering and updating their agency's data, number of ongoing or planned projects and number of identical and similar projects)—are compiled and stored for monitoring.
Access: Rules and Management
While the POP DB was stored in a workroom of EUnetHTA intranet during JA 1, it is now hosted and technically maintained by the Deutsches Institut für Medizinische Dokumentation und Information (DIMDI) and organizationally managed (user and access rights) by the LBI-HTA. The POP DB can be accessed by means of http://eunethta.dimdi.de/PopDB/. From earlier open-access efforts to set up a database for ongoing and planned HTA projects, it was found that compliance with requests to share data was poor. Therefore, it was finally agreed in the EUnetHTA Plenary Assembly, in the Executive Committee and in WP7 meetings to work with a “give-and-take” access rule: Only those EUnetHTA Partners who share (or at least respond that they have no information to share) are allowed to enter the POP DB. The access is managed by the LBI-HTA and the EUnetHTA Secretariat so that access rights management is conducted quarterly, after requests to enter new projects or to check the status of older ones are made.
Additional Materials: Work Sheets, User Manual, Communication Statistics
In addition to the development of the POP DB training materials (a presentation and exercise work sheets), a user manual was written so that the user is not dependent on the developer or the webmaster. Furthermore, because the POP Database is intended to facilitate rather than coordinate collaboration, a checklist for collaboration was developed and distributed to motivate EUnetHTA Partners to reflect about different intensities of collaboration (see Table 1). The checklist is based on a brainstorming exercise conducted by WP7 partners.
Table 1. Checklist for Collaboration: Possible Ways of Collaboration Using the EUnetHTA Planned and Ongoing (POP) Database
Database Content
Within the 3 years of EUneHTA JA 1, eleven quarterly requests, respectively, reminders to update the POP DB were sent out: Between thirty-four (1st request) and forty-three (11th request) of fifty-four (2010) to fifty-seven (2012) HTA agencies from nineteen (1st request) to twenty-four (5th to 10th request) countries had been given access to the POP DB because they responded to or updated their entries. These figures correspond to a response rate (percentage of those who replied as a function of all partner agencies) of between 64.6 percent and 78.6 percent (see Table 2).
Table 2. EUnetHTA Planned and Ongoing (POP) Database Statistics
In total, an average number of 1,107 projects (between 896 projects in June 2010 and 1,267 projects in November 2012) are ongoing/ planned within the EUnetHTA JA 1 partner agencies. Of those, approximately half are carried out in the UK by NICE or NETSCC (513), followed by CAHIAQ/ Spain (100), CVZ/ The Netherlands (91) and HAS/ France (53).
Between 10 and 12 percent (excluding the 1st request) of all the HTA production from the EUnetHTA JA 1 partners was identified to be identical (same technology, same indication) and 30–39 percent (again excluding the 1st request) were listed as similar topics with, for example, different research questions or approaches (see Table 3).
Table 3. Identical or Similar Projects in January 2013 (11th Request) in the EUnetHTA Planned and Ongoing (POP) Database
DISCUSSION
The POP DB has become an online database (see Figure 1) that hosts not only the titles and MeSH categorizations of planned and ongoing projects, but also some basic information on the status and contact details of the listed projects of EUnetHTA Partners. Currently, it stores more than 1,200 planned, ongoing or recently published projects of forty-three EUnetHTA Partners from twenty-four countries. Because the POP Database aims to facilitate collaboration, it also provides a matching system to assist in identifying similar projects. Overall, more than 10 percent of projects in the database are identical both in terms of pathology (indication or disease) and technology (drug, medical device, intervention). In addition, approximately 30 percent of all stored projects are similar, meaning that they have at least some overlap in content.
The POP DB is now a routine tool that supports the flow of information on new medical interventions. Every WP7 partner has contributed at least one entry to the system. EUnetHTA Partners keep their database entries updated; nevertheless, quarterly reminders are still sent to them. The “alerting” service on duplicate activities within EUnetHTA partner agencies is still being carried out manually, but this function is planned to become automated in the next version (POP DB v2). Opportunities for information exchange among agencies and collaboration on medical interventions are definitely facilitated by the POP Database.
Whether a reduction of duplication is actually taking place depends on the definition of collaboration: Anything from “wait and see until publishing” (of the other agency's report) to a full collaboration by producing a common report (see Table 1) is possible. The survey on the role of the POP Database in reducing the duplication of efforts—conducted at the end of JA 1 by the Centre for Applied Health Services Research and Technology Assessment (CAST, Denmark, see References)—revealed a straightforward reduction of duplication on the report level, as well as a partial collaboration with an exchange of information. All in all, twenty-three collaborations were reported, twelve of which were self-initiated by the LBI-HTA (Reference Huic, Nachtnebel, Zechmeister, Pasternak and Wild13). (Only) a slight reduction of work can be documented. Further evaluations on actual collaborations have to follow. These evaluations might focus on differences of collaborations for identical or similar projects and whether the definition of “similar” based on technology and indication is sufficiently sensitive. If the purpose of the POP Database is to facilitate collaboration, then efficient identification of projects and partners where collaboration might be useful or appropriate and devising methods to identify those links must be a priority.
One might ask why “only” around three-quarters of the EUnetHTA Partners respond to POP requests: The answer lies in the Joint Action construction itself. Many countries, especially those with little or no HTA production (Slovakia, Czech Republic, Cyprus, Bulgaria, Greece), are advocated by official ministry representatives and not by HTA-units: They are the non-responders. For the HTA-producing agencies, the compliance to update the POP DB is high and one is inclined to conclude that the “give-and-take” access rule works well.
In a POP DB v2 that will be released during JA 2 (2015), further functionalities are under development and are being integrated into the system. Those are: temporary save and later edit of a record, improvement of the search engine, semi-automatic reminder to update the database, alerts (e-mail notifications), CSV (Comma-separated values) export and interoperability with the HTA DB in CRD. Ideally, the records of published projects should then move to the HTA DB. This interface has been given the highest priority.
There are still some major issues that have to be tackled concerning POP DB. One of those to be solved is the inclusion of the many small—mostly academic—HTA-producing institutions into EUnetHTA activities, especially in the use of and access to the POP DB. The other issue is the opportunity as a consequence of the POP DB: The active coordination of collaboration based on the knowledge of work plans by means of calls for collaboration and other forms of active brokering of projects to reduce duplication. This would mean a more centralized approach, rather than leaving any activity to the individual agencies. Last but not least, EUnetHTA partners need to be motivated to enter their new projects in the database when they are still in a planning phase and not to wait until the projects are ongoing. This would increase the potential for collaboration.
CONCLUSIONS
The POP DB shows that there is a substantial topic overlap among European HTA producers. The major difference between other databases including ongoing (PROSPERO [7–Reference Booth, Clarke and Ghersi12], INAHTA [5,Reference Hailey6]) rather than solely published projects (DARE, EED, HTA [1,Reference Booth, Wright and Outhwaite14]) is that the POP DB took the carrot-and-stick approach by only providing access to those agencies who share their information. Although the POP DB is successful as far as regular updates of most national HTA agencies within EUnetHTA, little is known about its actual effects on collaborations in Europe. Devising methods to actively support collaborations such as project brokering and project management are of utmost importance as the next steps to be taken.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1 http://dx.doi.org/10.1017/S0266462314000567
CONTACT INFORMATION
Claudia Wild, Dr.phil. (claudia.wild@hta.lbg.ac.at), Director, Ludwig Boltzmann Institute of Health Technology Assessment (LBI-HTA), Garnisongasse 7/20, 1090 Vienna, Austria
Judit Erdös, MA (judit.erdos@hta.lbg.ac.at), Research Assistant, Ludwig Boltzmann Institute of Health Technology Assessment (LBI-HTA), Garnisongasse 7/20, 1090 Vienna, Austria
Marisa Warmuth, MD, MIPH, Senior Researcher, Ludwig Boltzmann Institute of Health Technology Assessment (LBI-HTA), Garnisongasse 7/20, 1090 Vienna, Austria
Gerda Hinterreiter, Mag., Ludwig Boltzmann Institute of Health Technology Assessment (LBI-HTA), Garnisongasse 7/20, 1090 Vienna, Austria
Peter Krämer, Dr. (Peter.Kraemer@dimdi.de), Deutsches Institut für Medizinische Dokumentation und Information (DIMDI), Waisenhausgasse 36-38a, 50676 Köln, Germany
Patrice Chalon, MSc (patrice.chalon@kce.fgov.be), Knowledge and Information Systems Manager Belgian Health Care Knowledge Centre (KCE), Boulevard du Jardin Botanique 55, 1000 Brussel, Belgium
Ludwig Boltzmann Institute for Health Technology Assessment (LBI-HTA)
CONFLICTS OF INTEREST
All authors report grants to their institutions from the European Union.
