As part of the European Network for Health Technology Assessment (EUnetHTA) Joint Action Project Framework (2010–12) it was envisaged to develop and test tools to avoid duplication of HTA activities by promoting collaboration between HTA agencies. Increased collaboration may reduce resource inputs for individual agencies, thereby freeing up resources for additional assessments. Reduction of duplication would also be helpful in justifying the resources spent on HTA at a global level. One of the tools to achieve collaboration was the Planned and Ongoing Projects (POP) Database collecting information on future and ongoing projects from EUnetHTA partners.
The POP Database is a multifunctional electronic database enabling searches by topic or agency. Identical or overlapping projects are automatically identified by means of a categorization system. Information in the Database is restricted to EUnetHTA partners who are quarterly requested to update information on project titles, MeSH categorization, project status, and contact details. By the end of the EUnetHTA Joint Action, the Database stored more than 1,200 projects of forty-three EUnetHTA partners from twenty-four countries. Overall, 10 percent of these projects were identified to be identical in terms of both pathology/indication/disease and technology/drug/medical device/intervention (Reference Wild, Erdös and Warmuth1;Reference Sandvei, Vondeling and Olsen2). The Ludwig Boltzmann Institute of Health Technology Assessment (lead institution) in Austria had the responsibility for the development and management of the Database. The Haute Autorité de Santé (HAS) in France served as co-lead.
A second major task of the lead institution was to support collaborative initiatives between agencies. This included among others developing a checklist for potential collaboration possibilities, contacting EUnetHTA partners with identical or overlapping POP projects (so-called “calls for collaboration”), as well as organizing workshops on specific topics with a high potential for collaboration. On this basis, a total of twelve inter-agency assessments were produced during the EUnetHTA Joint Action (Reference Sandvei, Vondeling and Olsen2).
The current article can be read as a stand-alone contribution. However, the article mainly builds on the work by Wild et al., providing an overview of the development, content, procedures and statistics of the POP Database, as reported in this special issue (Reference Wild, Erdös and Warmuth1). The analysis was carried out by the Centre for Applied Health Services Research (Reference Sandvei, Vondeling and Olsen2) (CAST) at the University of Southern Denmark. The main research question was whether the POP database has served its intended purpose of contributing toward reducing duplication of assessment efforts.
METHODS
We conducted an online survey (SurveyXact®) in December 2012 directed at all fifty-seven EUnetHTA partners that had received LBI-HTA's quarterly requests to submit information to the POP Database. Survey questions were developed based on the responses to two semi-structured telephone interviews with key personnel at the lead institution and content analysis of detailed information on the activities of the POP Database provided by the lead institution. The two interviews with the assistant to the director and the director of the lead institution, carried out in October 2012, lasted 30 minutes each. They provided us with background information on the development and management of the Database. The detailed documentation material developed by the lead institution included, among others, ten communication protocols, material produced in four face-to-face meetings, and the results from three surveys carried out by the lead institutions during the project. These surveys were directed at EUnetHTA partners that had provided information to the POP Database and focused on various themes, including users’ requirements to the Database (Survey 1), users’ impressions of the online Database (Survey 2), prioritization of Database developments and collaborating activities between agencies facilitated by the Database (Survey 3). While this combined data gave us detailed information on the content and structure of communication activities as well as on the statistics of the POP Database, partners’ actual use or non-use as well as resulting collaborations between agencies was less well documented. For instance the third survey focusing on collaborative activities between EUnetHTA partners had quite a low response rate with only seventeen of forty-two responding partners (40 percent). Therefore, to give a qualified answer to our research question, additional data had to be collected.
Our survey contained eleven multiple choice items and six free-text items and was organized into four main themes. This included use/non-use of the POP Database, collaboration between agencies without involvement of the lead institution, pros and cons of collaboration and reduction of duplication of work. We were particularly interested in collaborative activities that had occurred without involvement of the lead institution, as this would be most representative of the “real world”. One follow-up telephone call was directed at non-responders. The full survey questionnaire can be found in the project report (Reference Sandvei, Vondeling and Olsen2).
RESULTS
Providing Information to the POP Database
Forty-two of fifty-seven agencies (74 percent) responded to the survey. Forty of the forty-two responding agencies (95 percent) responded they knew the POP Database. Thirty of the forty-two responding agencies (71 percent) provided information to the POP Database.
Collaboration between Agencies
We asked agencies whether they had contacted other agencies because of information in the POP Database, that is without involvement of the lead institution, and of its frequency of occurrence (often, sometimes, seldom, never, do not know). Fourteen agencies out of the group of agencies which had indicated to know the POP Database responded that they had contacted other agencies (35 percent). One of these fourteen agencies (7 percent) replied that they had often been contacted, five agencies (36 percent) responded that they sometimes had been contacted, and eight agencies (57 percent) replied that they only seldom had been contacted.
Agencies were also asked if they had been contacted by other agencies because of the POP Database. Eleven agencies out of the group of agencies which had indicated to know the POP Database responded that they had been contacted by other agencies (28 percent). Four of these eleven agencies (36 percent) replied that they had sometimes been contacted, and seven agencies (64 percent) replied that they only seldom had been contacted.
We understand that eleven collaborations were started without the involvement of the lead institution as a result of these contacts.
Pros and cons of Collaboration
In two separate free-text questions, we asked agencies to share their experience on helpful and negative factors for collaboration. Twelve agencies out of the group of agencies which had indicated to know the POP Database shared their experience on collaboration (30 percent). Nine out of these twelve agencies (75 percent) considered the POP Database a useful tool for sharing knowledge, experience and information. Five agencies (42 percent) expressed that the POP Database provided an important first hand overview of ongoing HTA activities at a European level as well as an easy access to other agencies. Two agencies (17 percent) indicated that shared formulations increased both the power of and confidence in recommendations. Finally, two agencies (17 percent) stated that they had the impression that collaboration had increased efficiency in the work process.
On the negative side, agencies expressed that different time-windows (92 percent), different scopes (58 percent), and different working languages (58 percent) often limited opportunities for collaboration. Finally, two agencies (17 percent) stated that they had the impression that collaboration sometimes increased the workload, for example due to extra communication with an additional partner.
Reduction of Duplication of Work
A straightforward reduction of duplication at the report-level was reported twice. Collaborative activities were usually oriented toward information exchange on for example literature search protocols, extraction tables, info on other core elements than safety and effectiveness, executive summaries and full project reports. In one case, one respondent answered that a planned assessment was cancelled after having been informed by means of the POP Database that there were similar plans in another agency.
DISCUSSION
The data supplied to us by the lead institution were of high quality and the response rate to the survey was satisfactory. We believe that the combination of collected data has given us detailed insight into the usage and benefits of the POP Database. The results of the project may not be easily replicable though, given the Database's functioning as part of a European study.
Our study showed that collaboration between agencies and the ensuing supposed reduction of duplication is less straightforward than initially thought. This is, among other factors, due to the use of a local or regional language rather than an international language, the scope of an assessment, and its particular timeframe. A study by Huić et al. (Reference Huić, Nachtnebel, Zehmeister, Pasternak and Wild3) documented that time constraints and application of strict (time-consuming) methodology add to the problem. The overall picture that emerges is one of a sophisticated Database in an environment that may still have to do a lot of learning to use the facility intensively.
The barriers toward collaboration are legitimate and perhaps more serious than expected, but can be addressed. For instance, with regard to the choice of language, it is recommended that agencies write assessments in English, in addition to or replacing the local language. With regard to scope, HTA agencies could agree upon a minimum scope of every assessment. Furthermore, the HTA adaptation toolkit, which fifth version was produced in October 2011, could contribute to streamlining procedures and thereby reduce the problems associated with current practice (Reference Turner, Chase and Milne4).
How can the findings be interpreted when taking into account the wider HTA and EUnetHTA context? For the case of the POP Database, two databases seem to stand out: The PROSPERO Database, developed by the International Network of Health Technology Assessment Agencies (INAHTA), and the CRD-HTA Database, developed at the Centre for Reviews and Dissemination (CRD) at the University of York. PROSPERO is an international database of prospectively registered systematic reviews in health and social care (5). The CRD-HTA Database provides free access to bibliographic information about ongoing and published health technology assessments commissioned or undertaken by members of INAHTA and other HTA organizations around the world (6). All of these efforts to set up a database for ongoing and planned projects are driven by the intention to reduce duplication and release precious resources for an increased output of different rather than overlapping research results (Reference Wild, Erdös and Warmuth1). There are, however, several important differences between the initiatives. One of the major differences between PROSPERO and the CRD-HTA Database on the one hand and the POP Database on the other hand is that the latter is not publicly available. Moreover, of the three databases the POP Database seems to be the only one that supports collaboration between its potential users (5;6). We have the impression that these active elements of the POP Database are of key importance for its success.
CONCLUSIONS
At the end of the EUnetHTA Joint Action Project Framework (2010–12) there were more than 1,200 projects in the Database, 10 percent of which were identified to be identical. This indicates a potential to reduce duplication of effort, which has not been realized in the current project given the relatively small number of agencies involved in collaborative activities and the limited scope of collaborative efforts. We cannot base definitive conclusions on the basis of our findings. Reasons include the small sample and the truly preliminary experience that agencies have gained. A more definitive conclusion would among other factors require a more rigorous design. Further evidence needs to be gathered to determine whether the POP Database is effective and whether the benefits outweigh the resources required to maintain it. The scarcity of resources and the need for efficient delivery of HTAs suggest that collaboration will become increasingly important. It will require open-mindedness, explicit attention and a long-term perspective to gradually streamline the current fragmented process of collaboration between HTA agencies in the European Union. We hope that the issues raised in this study will be addressed in EUnetHTA Joint Action 2.
CONTACT INFORMATION
Hindrik Vondeling, PhD, Associate Professor of Health Technology Assessment (hvondeling@health.sdu.dk), Centre for Applied Health Services Research (CAST), University of Southern Denmark, J.B. Winsløwsvej 9B, 1th, 5000 Odense C, Denmark
Marianne Sandvei, MA, Consultant (masa@cast.sdu.dk), Centre for Applied Health Services Research (CAST), University of Southern Denmark, J.B. Winsløwsvej 9B, 1th, 5000 Odense C, Denmark
CONFLICTS OF INTEREST
Both authors report they have nothing to disclose
