Information is an essential instrument for decision making at the different levels of management in health. Currently, the use of scientific information in medicine runs into a serious problem related to the “explosion” of biomedical information. The biomedical information industry (editorials, scientific journals, and so on) does not respond to the supply and demand law. This dramatic growth in information has mainly been caused by the importance of publications for professional résumés and the proliferation of medical journals, as well as increased funding by both governments and pharmaceutical companies that has boosted research and increased the number of publications.
There are many sources where interesting biomedical information can be searched (Reference Bartkowiak1;Reference Grandage, Slawson and Shaughnessy9). Apart from the biomedical journals and databases, there are other resources such as monographs, text books, and bibliographic catalogues; gray literature (dissertation abstracts, congresses, ongoing clinical trials. . .); medical press (newspapers, gazettes. . .); the Internet (Reference Steinbrook16); health directories and portals; information from the Health Technology Assessment agencies (Reference Parada14); Internet sources intended to inform patients’ families and patients’ associations, and so on. This makes it necessary to sieve and select the sources to consult, and to prioritize the order in which to access such sources, depending on their capacity to provide information relevant to our work (Reference Egües-Olazabal, Gutierrez-Ibarluzea and Arcelay-Salazar8). These problems are especially relevant for the professionals working in the documentation units in the Health Technology Assessment agencies.
The first step of drafting any Health Technology Assessment document is the bibliographic search. These searches should be systematic and comprehensive, especially in the context of evidence-based medicine or health technology assessment in health care (HTA), where information is retrieved to inform decisions. However, taking into account the information “explosion,” being exhaustive and rigorous becomes difficult and incompatible with finding a fast answer. There is a lack of systematization in all the different search strategies used for the same research question at different HTA units.
In fact, there is no consensus among HTA units or agencies concerning which resources should necessarily be searched when developing a review of the literature. There are two main obstacles to uniformity in this area, the first being that the possibilities for accessing databases and other resources differ between agencies, the second being the differences between the preferences of information specialists when selecting information resources. This means there are no clearly defined prioritization criteria for selecting sources of information. In some of the HTA agencies, there are specific written guidelines for prioritizing information resources (6;Reference Mejuto Martí13;Reference Romero, Reyes, Osuna, Pírez and Gozalbes15). However, as far as we know, these guidelines are not uniform, at least not among Spanish agencies, and the recommendations provided in these documents are not based on consultations or consensus but have been decided on the basis of other parameters, such as the own experiences of the documentation specialists, or just based on the accessibility to databases and other information resources.
Within the context of literature searching for the purposes of health technology assessment, a protocol is an explicit, structured process for tackling the task of gathering information. It outlines, in a detailed and transparent way, a logical set of steps to work through in the course of the search. Theoretically, another researchers could duplicate the search strategy and retrieve comparable results. The COSI (Core, standard, ideal) protocol from New Zealand is an approach for selecting the relevant information sources depending on the topic of research, the kind of study, the time framework, and the available human and technical resources (Reference Bidwell and Fabricious Jensen3). In addition to the COSI protocol, there are other proposals for selecting information resources (4–6;10;13;15). Nevertheless none of those approaches has been designed in an explicit or systematic way.
The rationale for the systematic search, as described in the COSI protocol (Reference Bidwell and Fabricious Jensen3) or other similar protocols (Reference Gutierrez-Ibarluzea and Egües-Olazabal10;Reference Mejuto Martí13;Reference Romero, Reyes, Osuna, Pírez and Gozalbes15) came from work by The Cochrane Collaboration and the evidence-based medicine movement, which drew attention to the bias inherent in using only published literature, and the need for common approaches to literature searching within HTA organizations. However, there is a lack of evidence to show what proportion of new documents would be discovered in an ideal search, and how much they add to, or challenge, the information already found in main sources. This question is of major significance, and there is currently much interest in investigating it. This could form the basis for another, related study.
Search protocols appear to constitute a good attempt to standardize and facilitate the documentation process. This is the main reason why a consultation about search protocols and a systematic approach to define them was performed by the documentation specialists of the Spanish Health Technology Assessment agencies.
OBJECTIVES
The main objective of the study was to consider the ideas and opinions of the Spanish Health Technology Assessment agencies in relation to health sciences search protocols. This consideration involved defining the main advantages and disadvantages of using search protocols, and evaluating the possible risks and benefits for the documentation process (this strategy corresponds to Methods A & B). The second aim was to develop a questionnaire to define a series of criteria that could aid in prioritizing the databases and information resources to be consulted for each research project (Method C). The third aim was to rank this list of defined prioritization criteria (Method D).
METHODS
Context, Participants, and Data Collection
The information specialists within the Spanish Health Technology Assessment Agencies (AUnETS) group were asked to participate in the first three stages (Methods A, B, and C) of the project. Ten information specialists agreed to participate.
In the international context, nineteen information specialists from the Health Technology Assessment International's Information Resources Group (HTAi IRG) group collaborated in the final stage of the project (Method D). A flowchart summarizing the methodology of this study can be found in Figure 1.
Method A: Survey about Search Protocols
A survey was developed to evaluate experts’ opinions in relation to the usefulness of bibliographic search protocols. The survey was sent to the information units of the various AUnETS agencies. Six different areas were explored; the first area was related to the utility of using a search protocol when developing a bibliographic search versus consulting the main information sources (databases, journals, or other resources) and continuing with strategies such as tracking from electronic references, being alert and asking around, or “snowballing” from the reference lists of the main articles. The second area explored was the transparency and repeatability of search protocols. The third area focused on discussing the flexibility of a search protocol and its capacity to fit properly into different thematic areas and situations. The fourth area was related to the usefulness of search protocols as a way of keeping the documentation research stage of projects within a defined time limit, by specifying a limited number of sources to consult (ie, the top n sources as ranked by the search protocol), depending on the time available for research. The fifth area attempted to evaluate experts’ opinions about language bias caused by the limitations of using language in a standardized search protocol. The last area attempted to determine the importance of using experts’ opinions (topic specific experts (cardiology, mental health, and so on), documentation experts, or methodology experts such as epidemiologists) when deciding on the sources to be accessed or defining the priority of accessing those sources.
Method B: SWOT Technique
A consensus group meeting was organized to further explore the advantages and disadvantages of using bibliographic search protocols. In total, ten information specialists participated; the discussion technique used during the meeting was SWOT analysis (strengths, weaknesses, opportunities, and threats). The participants were asked to outline the strengths, weaknesses, opportunities, and threats of bibliographic search protocols. All the responses were written down.
Method C: Nominal Group Technique
During the same consensus meeting, a rough draft of a questionnaire about prioritization criteria was presented. This questionnaire had previously been developed by the documentation specialists working at the Basque Office for Health Technology Assessment. The objective was to discuss the draft using the nominal group technique to come up with a final version of the questionnaire.
The draft questionnaire was divided into five areas of prioritization criteria: study topic, characteristics of the database, characteristics of the database interface, characteristics of the organization, and expert opinion. The nominal group technique was divided into two phases. During the first phase, the information specialists were asked to come up with ideas in five areas: modification of prioritization groups; the impact of the study topic on which information resources to access; the characteristics of the databases; the characteristics of the database interface; the human, material, and time resources of the organization. There were two rounds per area and each participant was allowed to present just one idea per round; all the ideas were noted down. During the second phase the participants were asked to vote on all the registered ideas, ranking them in order from one to ten.
Method D: Prioritization Criteria Questionnaire
The final version of the prioritization criteria questionnaire designed during the nominal group technique meeting was e-mailed to the experts of the HTAi IRG group.
Data Analysis
Analysis was carried out on the transcript of the responses from the survey, the SWOT analysis, and the nominal group technique. The data were processed in a qualitative manner. All the information experts from the AUnETS group took part in the analytical process and discussed the results.
Data derived from the survey were summarized as advantages and disadvantages of using search protocols. The results were consequently sent to the participants for any possible further input, which was also assessed and included in the analysis to improve its validity. The English translation of the survey was revised by the participants to ensure the adequacy of the translation of the verbatim.
The responses from the SWOT meeting were summarized in four areas: strengths, weaknesses, opportunities, and threats of using search protocols. When similar ideas or concepts appeared in one of these four areas, they were combined to produce a synthesis of the information.
Figure 1. Methodology flowchart. AUnETS, Spanish Health Technology Assessment Agencies; HTA, health technology assessment; SWOT, strengths, weaknesses, opportunities, and threats; HTAi IRG, Health Technology Assessment International's Information Resources Group;
For the purposes of analyzing the data from the nominal group technique, a cutoff point of thirty was set. All the ideas with a score greater than thirty were taken into account and incorporated into the final version of the prioritization criteria questionnaire intended for the HTAi IRG.
RESULTS
Results Derived From the Survey About Search Protocols (Method A)
The results obtained from six heads of the information units of the Spanish HTA agencies that completed the survey about search protocols are summarized below:
Six different areas were explored; the first area was related to the usefulness of using a search protocol when developing a bibliographic search versus nonsystematic approaches (such as tracking from electronic references). Three of the six experts considered search protocols more useful compared with other possibilities. However, the other three participants considered that the utility of search protocols depends on the situation.
The second area explored was the transparency and repeatability of search protocols. All of the experts agreed that search protocols are transparent; however, their opinions differed regarding repeatability. Two experts thought that protocols were transparent but not repeatable. Four of the experts thought that search protocols were both transparent and repeatable.
The third area focused on discussing the flexibility of a search protocol and its capacity to suit different thematic areas and situations. Four of the experts expressed the opinion that a search protocol is flexible enough (or at least it that should be) to be adapted to the different research topics and situations. Another expert thought that it should be adaptable to different topics (by being able to search different databases, using inclusion and exclusion criteria, and so on) but not to different situations. Finally, there was one expert who thought that it would be very difficult for a standardized protocol to be adaptable to all situations.
The fourth area was related to the usefulness of search protocols for helping to stay within established time frameworks for the documentation research stage of projects. Four of the participants did not consider that protocols were capable of fixing time frameworks. Another participant stated that he considered protocols necessary to be able to fix time frameworks, and the last one simply expressed the opinion that it might indeed be useful to talk about the number of hours spent on searches.
The fifth area attempted to evaluate experts’ opinions in relation to the possible bias resulting from using language (i.e., key words) as a search criterion in a standardized search protocol. Five of the experts expressed the opinion that language is not a good variable for limiting searches. One expert said that English should be the search language.
The last area attempted to determine the importance of using expert opinion (topic-specific, documentation, or methodology experts) when deciding which sources should be accessed or defining the priority of accessing these sources. The vast majority of participants thought that experts’ opinions in different areas were always a good basis on which to develop an information search.
Results Derived From the SWOT Technique (Method B)
After the meeting with the AUnETS group, the strengths, weaknesses, opportunities, and threats of using search protocols were determined (see Table 1).
Results Derived From the Nominal Group Technique Used During the Consensus Meeting (Method C)
The final version of the prioritization criteria questionnaire was designed based on the votes of the consensus meeting. This final version included five areas: study topic; characteristics of the database and other information resources; database interface; characteristics of the organization; kind of research output for which the information is intended. Twenty-one of fifty-eight prioritization criteria scored higher than the cutoff point of thirty and were assigned to the five areas mentioned above (see Table 2).
Results Derived From the Prioritization Criteria Questionnaire (Method D)
The responses from the HTAi IRG group were pooled together and a ranked list of criteria, from the most to the least relevant with the mean punctuation obtained, is presented in Table 3.
DISCUSSION AND CONCLUSIONS
Some authors have stated that a structured protocol acts as a framework and guide to information searches. It lists all the information sources to be searched for the assessment and ensures that the search proceeds in a structured manner. The advantages of such a protocol according to the literature (Reference Bidwell and Fabricious Jensen3;Reference Gutierrez-Ibarluzea and Egües-Olazabal10;Reference Romero, Reyes, Osuna, Pírez and Gozalbes15) are consistency of information searching within the organization, ranking of sources within established time frameworks, flexibility to be adapted to different local situations, dynamicity, flexibility to be adapted to each individual topic, appropriateness of the sources to be searched, determination of agreed cutoff points within an organization, and transparency of the search process.
Wanke et al., 2006 (Reference Wanke, Juzwishin, Thonley and Chan17) recently reported on an evaluation of HTA documents from different organizations on an international scale and observed a substantial level of heterogeneity between them. This kind of heterogeneity also exists between the information searches performed for different HTA documents about the same topic and with the same purpose, for which different search strategies are reported. A common approach and consensus between different organizations is necessary to make core information searches about the same topics and in the same areas as similar as possible.
In the present report, we have tried to define explicitly the advantages and disadvantages of using search protocols as assessed by the AUnETS group. The information specialists consulted agreed that search protocols were useful tools for guiding systematic searches. Some of the advantages defined were systematization of the searches and transparency and repeatability of the process. On the other hand, the disadvantages identified were inflexibility to be adapted to some situations, inability to establish time frameworks, and the difficulty of introducing experts’ opinions into closed protocols.
Table 1. Results Derived from the SWOT Technique
SWOT, strengths, weaknesses, opportunities, and threats; HTA, health technology assessment.
When comparing our results with previous findings, we observed general agreement in terms of the advantages mentioned above, that is, systematization, repeatability, and transparency of the search process. Nevertheless, we identified disagreement in the following areas: flexibility to adapt to different situations and capability to establish time frameworks. The Spanish experts also pointed out that implementing protocols could be difficult due to the reluctance of some HTA organizations to accept them and the impossibility of accessing some important information resources requiring payment or passwords.
However, as defined by the opportunities and strengths reported in our results, search protocols can be considered as valuable tools for guiding systematic information searches. In fact, protocols facilitate the work of searching, making the whole process more efficient; incorporate the work of information specialists; and facilitate the work of the people involved in HTA, making it easier for those less trained in information skills. The joint work carried out by the HTA agencies and networks seemed to be a strength that could be used to develop well-designed protocols, which would provide better management of the excess of information, save time and in general, provide more accuracy in the search process.
Other experiences (2;7;11;12) indicate that, depending on the topic and purpose of the assessment, different cutoff points in the search process will probably be necessary. Similar conclusions have been found in our research; in fact, time frameworks and cutoff points for the search process should be established by information specialists.
Table 2. Results Derived from the Nominal Group Technique
Table 3. Ranked List of Prioritization Criteria (Mean Value)
Our research has been the first attempt worldwide to establish explicit criteria for prioritizing information sources. The twenty-one identified criteria were assigned, based on the nominal group technique, to five areas: study topic; characteristics of the database and other information resources; database interface; characteristics of the organization; and kind of research output for which the information is intended. These criteria have been ranked in order of importance, according to the answers of the HTAi IRG group. In the near future, during a final phase of this project, the prioritization criteria and the information resources will be cross-linked to establish which information sources should be consulted. Our study shows one way in which a more uniform approach to the search process is being developed by the Spanish HTA agencies.
POLICY IMPLICATIONS
The present research project shows the need for consensus and a common methodological approach in the first step of drafting an HTA document, which is the bibliographic search. This has been the first systematic approach to determine the prioritization criteria to be taken into account when deciding which sources of information should be searched. These criteria will be used to define the minimum shared core databases that should be searched systematically by Spanish HTA agencies when producing mini-HTAs, full-HTAs, and clinical practice guidelines. Based on these criteria, similar approaches could be followed up in other contexts.
CONTACT INFORMATION
Itziar Guemes-Careaga, MS (osteba-san@ej-gv.es), Health Technology Assessment Technician, Department of Health, Osteba-Basque Office for HTA, Donostia-San Sebastian, 1, Vitoria-Gazteiz 01010, Spain
Iñaki Gutiérrez-Ibarluzea, MS, PhD (osteba7-san@ej-gv.es), Associate Professor, Department of Biochemistry and Physiology, Nursing University School Vitoria-Gasteiz, Jose Atxotegi z.g., 01009 Vitoria-Gasteiz; Health Technology Assessment Technician, Osteba-Basque Office for HTA, Health Department—Basque Country, Donostia-San Sebastian, 1, 01009 Vitoria-Gasteiz, 01010 Vitoria Gasteiz, Spain
The authors acknowledge the contribution of the information specialists of the Spanish Health Technology Assessment Agencies (AUnETS) group: Maria Sobrido, Raimundo Alcazar, Montse Salas, Mercedes Reza, Beatriz Duque, Antoni Parada, Victor Sarmiento, Antonio Romero, and Teresa Mejuto for their help, critical review, and invaluable comments on a previous version of the manuscript. Also thanked are the members of the HTAi IRG who agreed to answer the prioritization criteria questionnaire: Nina Hakak, Barbara Niedzwiedzka, Suzy Pasley, Pierre Vincent, Robert Aubin, Karen Rydin, Noertjojo Kukuh, Sue Bayliss, Sara Vanstone, Beata Gouba, Malene Fabricious, Grahan Riita, Julie Glanville, Lian Thye, Price AM, Mary-Dough Wright, Anne Fry Smith, Deborah Marshall & Janice Varney. The research was partly funded by a research project commissioned by the Spanish Quality Agency of the Spanish Ministry of Health.
