The European network for health technology assessment (EUnetHTA) was founded in 2006 with strategic objectives to promote more effective use of resources, increase HTA input to decision making, strengthen the link between HTA and policy making, and support countries with limited experience in conducting HTAs (Reference Kristensen1). Since 2009, EUnetHTA partners have had access to four Web sites or online databases (also referred to as “tools”) created during the EUnetHTA Project period (2006–08): the Members-only site (MO site), the Planned and Ongoing Projects Database (POP DB) (Reference Warmuth2), the HTA Core Model® online, and the EVIDENT database (formerly known as EIFFEL) (Reference Quentin, Carbonneil and Moty-Monnereau3). These tools were developed independently, without specific coordination or vision concerning interoperability. This resulted in tools with a different layout, vocabulary to index content and login systems which raised some complaints amongst end users.
Between 2010 and 2012, the work of EUnetHTA was organized into eight work packages (WPs) and financially supported by the Health Program of the Executive Agency for Health and Consumer of the European Commission Joint Action (JA) program (see the EUnetHTA Web site for more details) (4). The activity “Work package 6 - Information Management System” (WP6), led by the Belgian Health Care Knowledge Centre (KCE), brought together seventeen partners from fifteen countries (see list of partners in Table 1). The main objective of this work package was to provide a contemporary Information Management System (IMS) within EUnetHTA, ensuring compatibility and interoperability of tools across work packages to facilitate collaborative HTA work, and rapid dissemination of HTA results to all EUnetHTA partners (see list of tools in Table 2). The activities of WP6 were defined in a 3-year work plan and included: (i) further development of tools created during the 2006–08 period to enhance homogeneity and facilitate their usage by end users; (ii) development of new tools, including demonstrators of well-established standards that could support the general objective of rapid dissemination of HTA results; and (iii) identification of already existing tools developed outside of EUnetHTA to avoid duplication of efforts, and evaluate opportunities of collaboration.
Table 1. Partners of the EUnetHTA Work Package 6 Information Management System (2010–2012)
*Merge of ESKI and EMKI.
**Now Zorginstituut Nederland (National Health Care Institute) –ZIN.
***AReSS has ceased its activities in 2013.
Table 2. EUnetHTA Tools of the EUnetHTA Information Management System (2010–2012)
The objectives of this article are to describe: (i) how WP6 group was created and managed to facilitate the identification and agreement on common standards for EUnetHTA tools; (ii) how interoperability standards could be demonstrated; (iii) the status of collaboration with information systems outside EUnetHTA reached at the end of the 3-year period.
METHODS
Agreement on Common Standards
A mixed group was created by the EUnetHTA Joint Action work package 6 coordinator: one representative of each EUnetHTA tool team (HAS for EVIDENT DB, LBI-HTA for POP DB, NETSCC for Adaptation Glossary, THL for HTA Core Model® online) and (potential) users of the tools were invited to take part in WP6. Four face to face meetings were planned (two in year 1, one in year 2, and one in year 3); organisation and participation (travel, hotel) costs linked to those face to face meetings were funded by the Joint Action. To facilitate a sense of belonging to the group and favor participants engagement and communication between technical and nontechnical people, two ice breaking techniques were used at the first face to face meeting in April 2010 as replacement of the traditional round of presentations. Participants were first tagged with Post-its® of different colors related to their profile (IT or non-IT), and their experience in EUnetHTA (having previously participated to EUnetHTA or not). Then, three rounds of “speed dating” were organized: at each round, attendees were asked to choose a partner with another profile (to get pairs of IT and non IT); and talk for 3 minutes to exchange information about their respective work. Oral feedback from participants was requested at the end of the exercise, and during the last face to face meeting at the end of the 3-year period and summarized in the meeting minutes. Potentially interesting “standards” were identified by WP6 lead (KCE) and co-lead (DIMDI) by searching the Web, the literature, or by consulting colleagues. A synthesis document was prepared to support group discussions. Partners met face to face three times to discuss and agree on common standards for the development of the tools. Agreement was reached by consensus (see Table 3); implementation was under the responsibility of the respective work packages (WP4 for HTA Core Model® Online; WP6 for POP DB, Adaptation glossary online, Aggregators; WP7 for EVIDENT DB).
Table 3. Standards Selected by WP6
Demonstration of Interoperability Standards
Among the tools developed as part of WP6, two Web sites aimed at demonstrating widespread interoperability standards: the “Repositories aggregator” for the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) protocol, and the “News aggregator” for the RDF Site Summary (RSS) protocol. Sources to aggregate were identified by searching the partners’ Web sites, dedicated repertories, and by direct contact with colleagues. The “aggregator” Web sites were presented or demonstrated within WP6, at other EUnetHTA Work Package meetings, and also to a broader audience (e.g., EUnetHTA Conference, HTAi Conference).
Collaboration with Information Systems Outside EUnetHTA
Tools already in use outside EUnetHTA were identified by searching the Web, the literature, scientific procedures, and by consulting colleagues. The developers identified through this process were subsequently contacted by e-mail to explore opportunities for collaboration; and in some cases, phone conferences or face to face meetings were organized.
RESULTS
Agreement on Common Standards
The first face to face meeting was held in April 2010. It brought together eighteen people from thirteen Partner's organisation. Among them, thirteen tagged themselves as new to EUnetHTA; ten as content specialists, five as IT specialists, and three as having both profiles. The first feedback, provided after the ice breaking session, was positive: participants reported that the speed dating exercise facilitated the exchange of relevant information about their respective activities, responsibilities and about EUnetHTA in a fast and humorous way. At the end of the 3-year period, participants reported that the group had benefited from a positive atmosphere; the ice breaking techniques conducted at the first face to face meeting and the role of the group moderator were reported as supporting factors. This good atmosphere was identified as an important element that helped each type of participant to present their specific perspective (developer versus user), and nontechnicians to ask for “translation” of technical concepts and “jargon” into mainstream terms to facilitate a better understanding (e.g., Single Sign On). Exchanges were also considered very fruitful for both types of participants: developers gained better insights on the needs of the users regarding IT tools, while users gathered a better view of some of the most common constraints encountered when developing IT tools. Four common standards types that tool developers should comply with were identified and agreed upon: metadata, layout, vocabulary and authentication (see Table 3). Subsequently these were described in a guidance document (Reference Chalon, Kraemer and EUnetHTA5). The Metadata Terms from the Dublin Core Metadata Initiative (6) were selected as a reference to organize the content of records and label their fields. When the Dublin Core did not fulfill the needs, additional metadata can be chosen, preferably from an existing metadata scheme already endorsed at the European level to be found at the SEMIC Web site (Reference Goedertier7). A common layout for EUnetHTA tools was defined based on the initial EUnetHTA public site (2006–08) and the EUnetHTA graphical guidance (Reference Allgurin Neikter8). This common layout includes a two-column structure with headers and footers. While standard components are defined for each area, several alternatives are provided for developers to suit the specific objectives of their respective tools. Partners agreed on that the Medical Subject Headings (MeSH©) (9) was the best suited vocabulary to describe topics of records. Where relevant, several fields can be provided (e.g., one field for the problem and one field for the intervention, respectively P and I from the PICO clinical question) (Reference O’Connor, Green, Higgins, Higgins and Green10). A link to the MeSH© database and the HON Select (11) which offers a free access to MeSH in several languages is recommended to help users to identify the appropriate term to search content from the EUnetHTA tools. While covering all the necessary domains, MeSH© was not universally used in other information systems of the participating countries. Therefore, developers were free to implement any supplemental vocabulary that would fit the needs of their tools (e.g., ATC codification, ICD-10 classification). The Lightweight Directory Access Protocol (LDAP) (Reference Sermersheim12) was selected as common standard for authentication based on its wide use and opportunities for rapid implementation. It allow users to access the different EUnetHTA tools with the same login details, called EUnetHTA ID. The Members Only site was used as a source of credentials for all tools. By the end of the 3-year period, all tools were modified or designed following the common standards previously described.
Demonstration of Interoperability Standards
The “Repositories Aggregator” demonstrator was delivered at the end of 2010. This tool was automatically aggregating the two institutional repositories available at the time of development: LBI-HTA (Austria) and KCE (Belgium). Visitors could thus browse or search the aggregated catalogue and get access to the full text of reports hosted at the source repository. Because no new repository could be identified at the end of the 3-year period; it was decided to close it down because of incompleteness. The second demonstrator, the “News aggregator”, was delivered in 2011. Initially, nine RSS feeds were included (three coming directly from the partner's Web site, six being created by a third party online service). Each RSS item was transformed into a Web page being automatically categorized by country, partner and language. The availability of RSS feeds was regularly revised, with twenty-two available in 2012 (of which ten were obtained through a third party online service); while thirteen partners’ Web sites still did not provided any RSS feeds. By the end of 2012, the News aggregator had received an annual average of approximately 1,000 unique visitors. From January 2014, visitors can browse or search within more than 10,000 aggregated news.
Collaboration with Information Systems Outside EUnetHTA
Additionally, four HTA tools developed outside EUnetHTA but of potential interest for EUnetHTA activities were identified. Their developers were contacted to evaluate options to collaborate. Contacts resulted in (i) direct collaboration (HTA glossary, HTAi Vortal), (ii) evaluation of options for interoperability between tools (CRD HTA database), or (iii) formal framework to prepare cooperation on concrete projects (INAHTA projects database) (see Table 4).
Table 4. HTA Tools Developed Outside EUnetHTA and the Related Collaboration Status
DISCUSSION
Projects bringing together people from a large variety of cultural and professional backgrounds, and making use of electronic meetings have been shown to be challenging in several contexts including Community of Practice initiatives and e-learning (Reference Fung-Kee-Fung, Boushey and Morash13;Reference So and Brush14). In such contexts, face to face interactions are often highly valued (Reference Kirkman, Rosen, Tesluk and Gibson15); and ice breaking activities are recommended in situations where a group of people who never met will have to interact (Reference So and Brush14;Reference Bonk, Olson, Wisher and Orvis16). The group of WP6 partners’ representatives was indeed heterogeneous and, being geographically dispersed, it could even be considered as primary “virtual” (Reference Powell and Single17). Meeting face to face and including an ice breaking session at the first meeting were perceived as important factors for establishing the positive group atmosphere reported in the feedback. Still, further evaluation would be necessary to explore to which extend those two approaches influenced the group atmosphere.
Discussions over common standards raised doubt about the ability of MeSH© to perfectly describe the content. Furthermore, some participants reported that those in charge of content provision lack familiarity with MeSH©. Although these two issues are quite common when using MeSH; this has proven to be adequate in several circumstances (Reference Lenner, Lundgren and Damber18–Reference Pereira, Neveol and Kerdelhue20), and the provision of support and training usually helps people to better perform in indexing information (Reference Maggio, Bresnahan and Flynn21). Nevertheless, further investigation would be needed to evaluate the MeSH© implementation into each tool and the support provided to encoders.
With Joint Action 1, EUnetHTA left the pioneering period for a stabilization period. This is also reflected by the evolution of its Information Management System: thanks to WP6 activities, all tools now look more similar and familiar to the end users (same layout, same vocabulary, similar meta data), and can be accessed by means of a single login system. All this should benefit the end user by providing more efficiency and memorability (as defined by Nielsen (Reference Nielsen22)), and thus enhancing satisfaction. While such an assumption has not been directly tested, it is generally supported by the results of the general surveys conducted once a year where 60 percent of respondents found single authentication useful or very useful, and where barriers to the use of tools related to tools themselves diminished over time (Reference Guegan and Cook23). Other international collaborations networks like the Cochrane Collaboration face similar challenges; and foresee similar approaches: users will be at the centre of tool developments in the forthcoming years, and interoperability between the different software is planned to be developed thanks to a new concept of information architecture (Reference Mavergames24).
Despite the interest shown during discussions regarding the added value of OAI-PMH (Reference Chalon25), the Repositories Aggregator was shut down in 2012. Barriers to implement an institutional repository are known to be human rather than technical (Reference Jakobsson26). Adapting the current publication flow, and providing the necessary supporting IT infrastructure are indeed challenges difficult to overcome in such short time period. Searching the HTA database and the Web site of agencies (possibly through the HTAi Vortal) will thus probably remain the standard at many agencies (Reference Vlayen and Chalon27;Reference Straus, Tetroe and Graham28) until some technical evolution offers new opportunities in the coming years (e.g., Web Content Management Systems used for the Web sites of partners being compliant to OAI-PMH or any other data linkage standard). Regarding RSS, while several Web sites do not directly provide RSS feeds, we can expect that the status will positively evolve because a majority of them are driven by Web Content Management Systems that offer such an option. Asking partners to activate this feature and provide a specific RSS feed for published reports is, therefore, currently considered to be a good option to support rapid dissemination of this information through a RSS aggregator. Additionally, this would offer the opportunity to disseminate the same information through other channels like widgets, social networks, browser toolbars, and mobile applications. The monitoring of partners’ Web sites and the development of the Aggregator will thus be pursued.
While collaboration was always positively perceived by maintainers of tools developed outside the EUnetHTA IMS, technical developments would require additional resources. The integration of interoperability standards in the respective development roadmap of tools should be the first step on the way to interoperability.
CONCLUSIONS
While being entitled a project on IT infrastructure, the EUnetHTA Work Package 6 “Information Management System” was also about people. On the one hand, we had participants with different backgrounds and experiences. The three years’ experience showed that when having to agree on complex topics, fostering a cohesive group dynamic and hosting face to face meetings brings added value and enhances understanding between partners. On the other hand, we had the end user of the tools. The adoption of widespread standards should help to provide a sense of “feeling at home” when using any of the EUnetHTA tools. For example, a single login facilitates access to the tools which should in turn enhance their use, and, therefore, contribute to the general objective of EUnetHTA.
Finally, WP6 also puts in place the basis for data linkage between EUnetHTA tools and tools developed outside EUnetHTA. For this linkage to happen, a clear and official mandate and appropriate funding, like the Joint Action mechanism, is essential when interoperability is expected to come into place in a short term. Without such a support, the coordination of development is more complex, even when all parties are willing to enhance global efficiency and reduce duplication of efforts. Still, interoperability between EUnetHTA tools and others could be achieved if expectations on the timing of implementation are adjusted. The initiatives on interoperability of systems started in the first Joint Action period will be developed further in the second Joint Action period (2012–15); and, we hope, will result in a general interoperable information system that goes beyond EUnetHTA to serve the whole HTA community.
CONTACT INFORMATION
Patrice X. Chalon, MSc (patrice.chalon@kce.fgov.be), Belgian Health Care Knowledge Centre (KCE), Blvd du Jardin Botanique, 55, B-1000 Bruxelles, Belgium
Dr. Peter Kraemer (Peter.Kraemer@dimdi.de) Deutsches Institut für Medizinische Dokumentation und Information (DIMDI)
CONFLICTS OF INTEREST
Both authors report grants to their institutions from the European Union.
