Authors of systematic reviews of adverse effects, as with reviews of effectiveness, tend to focus on searching MEDLINE and reference checking to identify relevant studies for inclusion (Reference Golder, Loke and McIntosh1;Reference Golder, Loke and McIntosh2). However, research which compares data sources for information on adverse effects indicates that MEDLINE may not yield the most data on adverse effects, particularly when searching for drug-related adverse effects (Reference Golder and Loke3). While omitting MEDLINE from searches may seem unthinkable, there may be major time and cost implications for review authors when deciding to use different data sources instead of, or beyond just MEDLINE.
For instance, a systematic review on the contribution and yield of different information sources has suggested that Derwent Drug File, EMBASE, and industry submissions may provide the highest number of relevant references or unique relevant references with information on adverse effects (Reference Golder and Loke3). However, these findings are limited by the restricted range of sources compared, and the failure to take search strategies into account. Moreover, the included methodological evaluations actually date back many years, and are of uncertain relevance to current practice. Hence, the objective of this case study was to determine the contribution of searching a diverse range of different sources to identify adverse effects data for a systematic review, taking into account any limitations of the search strategies.
METHODS
Case Study
A case study systematic review of thiazolidinedione-related fractures in patients with type 2 diabetes mellitus was carried out to be able to assess the contribution of different data sources.
This review was an update of a previous systematic review (Reference Loke, Singh and Furberg5).
To be able to assess the efficiency of using different sources to identify information on adverse effects, a wide range of sources were searched.
The search strategy was based on the intervention; thiazolidinedione (rosiglitazone and pioglitazone) and the outcomes; fractures or bone mineral density. The strategy was translated for each database and was kept as consistent as possible across databases in order that a fair comparison between database results could be made.
The included references from this case study systematic review formed the basis of the analysis. For the primary analysis, all publications which presented enough data to calculate the odds ratio, relative risk or weighted means difference for meta-analysis were included.
Analysis
Individual Assessment of Sources. A record was made of where each of the included references were available and where they were identified. For each reference available on a database but not retrieved by the search strategy, the bibliographic record was then examined to determine why it had not been identified. A record was also made of any relevant references identified or available from only one data source.
Minimum Combination of Sources. The minimum combination of sources required to identify all included publications using the search strategies used was recorded. In addition, the minimum number of sources from which all the included publications were available (independent of the search strategy used) was recorded.
Individual Study Identification. To allow for multiple publications for the same study, the analysis was repeated with all relevant individual studies (as opposed to all relevant publications).
RCTs and Observational Studies. The analysis was repeated with the included RCTs and observational studies separately because certain databases might provide better access to specific study designs, for example, CENTRAL focuses on clinical trials.
Marginal Sensitivity and Marginal Precision. The overlap in content between the sources in terms of relevant and non-relevant records is of importance. The additional relevant records retrieved and the additional non-relevant records retrieved from searching the sources in two particular orders were assessed. Order 1: A theoretical order beginning with the source from which the highest number of relevant records was retrieved, followed by the source from which the highest number of additional relevant records were retrieved and so forth until all the relevant records are retrieved. Order 2: An order of sources that reflects common practice in systematic reviews. The top ten most frequent sources were selected (in order of popularity) from a review of systematic reviews of adverse effects (Reference Golder, Loke and McIntosh1;Reference Golder, Loke and McIntosh2).
RESULTS
Records Retrieved
From the database searches 3,591 unique records were retrieved (5,663 before deduplication). An additional 680 records (before deduplication) were retrieved from searches for ongoing studies, 629 spontaneous case reports, 90 monographs or chapters from databases or texts, and 10 entries in databases or texts that list adverse effects.
Included Studies
Fifty-eight references (representing forty-one studies) were included in the systematic review.
Where Were the References Identified?
Bibliographic Databases. Using the search strategies with the “drug” and “fracture” terms retrieved at least one included reference in all the databases, except Inside Conferences (Table 1). The highest sensitivity was achieved from searching Science Citation Index (SCI) (60 percent), followed by BIOSIS Previews (47 percent), EMBASE (41 percent) and then MEDLINE (33 percent). In the majority of the databases, precision was relatively high in the context of systematic review literature searches (Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2012014).
Table 1. References (RCTs and Observational Studies) Retrieved, in Order of Decreasing Number of Relevant Records Retrieved
Note. NA (non-applicable) – in some instances, the source was browsed rather than searched, so THE number of references retrieved is not applicable.
* In order to be able to assess the effectiveness of searching Internet search engines only the first three pages of results of Internet searches were screened for relevant articles. Only the first three pages were selected in order to reflect common practice in searching the Internet (Reference Jansen4;Reference Watlington7;Reference Zwicky8) and due to the impractical nature of reviewing all the results from Internet searches which can often be millions or even billions of pages.
Minimum Combination of Sources to Identify All Relevant References. The minimum combination of sources to retrieve all the relevant references with the search strategies used in this case study was; GlaxoSmithKline (GSK) Web site, Science Citation Index (SCI), EMBASE, BIOSIS Previews, British Library Direct, Medscape DrugInfo, handsearching, reference checking, AHFS First, and Thomson Reuters Integrity or Conference Papers Index (CPI).
Individual Study Identification
A similar pattern emerged when limiting the evaluation of identified records to individual studies as opposed to the total number of publications (Supplementary Tables 1 and 2, which can be viewed online at www.journals.cambridge.org/thc2012014).
Where Were the References Available?
The majority of the searches (using fracture and drug terms) did not retrieve all the relevant references available on each database (Table 1). The only databases in which all the relevant references available were identified were either conference proceedings databases or the drug company database (Table 1). Due to the limitations of the interface for the drug company database, this database was searched with the drug terms only and all records sifted for the adverse effect. Almost all the records missed did not contain any terms for “bone” or “fracture” in the bibliographic details.
The majority of the references were available on the Internet by a search on the specific reference using search terms from its citation (Table 1). Those articles that were not available on the Internet tended to be conference proceedings. The bulletins, newsletters, and referenced or partially referenced sources were all handsearched, therefore the number of relevant references identified from these sources matches the number of relevant references available (Table 1).
Minimum Combination of Sources with Relevant References Available. The minimum number of sources that contained all the included references was Science Citation Index (SCI), Medscape DrugInfo, BIOSIS Previews, British Library Direct, and handsearching.
Where Were the RCTs Identified?
The databases which achieved a higher sensitivity when searching was limited to RCTs, as opposed to all studies, were CENTRAL, GlaxoSmithKline register, Thomson Reuters Integrity, and Medscape DrugInfo (Supplementary Table 3, which can be viewed online at www.journals.cambridge.org/thc2012014).
Where Were the Observational Studies Identified?
The most notable difference in sensitivity for observational studies as opposed to all types of studies was for Science Citation Index (SCI) which increased from 60 percent to 86 percent (Supplementary Table 4, which can be viewed online at www.journals.cambridge.org/thc2012014).
The number of missed references in almost all the databases was notably higher when searching for RCTs than observational studies (Supplementary Table 3 and 4).
Marginal Sensitivity and Marginal Precision
Order 1: Highest Number of Relevant References First. When the sources are searched in order of retrieval of the highest number of relevant records until all the relevant references are identified (Supplementary Table 5, which can be viewed online at www.journals.cambridge.org/thc2012014), the order for searching is as follows: Science Citation Index (SCI), GlaxoSmithKline (GSK) Web site, EMBASE, AHFS First, Handsearching, Conference Papers Index (CPI), British Library Direct, Medscape DrugInfo, BIOSIS Previews, Reference checking.
Although searching this combination of sources identifies all the relevant references (100 percent sensitivity), overall precision was low at 3 percent.
Order 2: Current Practice in Systematic Reviews. If only MEDLINE had been searched along with reference checking then only 34 percent (20/58) of the relevant references would have been identified. Even a search of MEDLINE, EMBASE, and CENTRAL along with reference checking, would have retrieved less than half (43 percent, 25/58) of the relevant references (Supplementary Table 6, which can be viewed online at www.journals.cambridge.org/thc2012014).
There were three references that would not have been identified had the search been restricted to the top ten most popular sources of data used in systematic reviews of adverse effects.
DISCUSSION
Our study demonstrates the value of searching multiple sources to identify adverse effects data for a systematic review. In this case study, the minimum number of sources which needed to be searched (with a proposed search strategy) to identify all the relevant references was ten. Even if it were possible to devise a perfect search strategy that could retrieve all the relevant references available on each source, a minimum of five sources would still need to be searched. The most common practice of searching just MEDLINE and reference checking would have failed to retrieve two-thirds of the relevant references (38/58, 66 percent). Even a search of MEDLINE, EMBASE, and CENTRAL along with checking of reference lists would have failed to retrieve over half the relevant references (33/58, 57 percent).
The high sensitivity achieved in Science Citation Index (SCI), BIOSIS Previews, and Scirus, in particular, warrant further investigation. Science Citation Index (SCI), and Scirus have not been included in other evaluations (Reference Golder and Loke3), and as this is only one case study, it would be difficult to generalize the findings to other systematic reviews without further research.
As with previous research (Reference Golder and Loke3) on identifying drug intervention studies, searches on EMBASE yielded more relevant references than on MEDLINE in this case study. Similarly, MEDLINE, EMBASE, and TOXLINE retrieved more relevant references than Iowa Drug Information Service (IDIS), As with previous studies (Reference Golder and Loke3), many unique studies were identified through contacting manufacturers or searching manufacturer Web sites. However, the ease of retrieving industry funded studies varies greatly depending on the drug company. Whereas previous case studies have indicated a higher yield from Derwent Drug File than MEDLINE or EMBASE (Reference Golder and Loke3), in this case study both MEDLINE and EMBASE retrieved a higher number of relevant references than Derwent Drug File.
The low sensitivity achieved by some of the databases is not surprising. For instance, specialist conference databases such as Conference Papers Index (CPI) and Inside Conferences contain only conference abstracts, CINAHL specializes in nursing and allied health, the GlaxoSmithKline Web site contains only industry funded studies and CENTRAL focuses on clinical trials. Non-bibliographic databases are typically aimed at drug developers or prescribers, and do not claim or intend to be comprehensive from a systematic review point of view.
In all the databases, the number of missed studies with the search strategies used was higher for RCTs than for observational studies. This may be because adverse effects are more likely to be a secondary outcome in RCTs than in observational studies and therefore adverse effects terms are less likely to appear in the title, abstract or indexing/keywords of the bibliographic records of RCTs than observational studies.
Of interest, the minimum combination of sources required to identify all the relevant references did not include MEDLINE and neither did the combination of sources identified through the selection of the sources with the highest sensitivity first. This is partially due to the fact that searching MEDLINE did not identify any unique references or have the highest sensitivity.
LIMITATIONS
The main limitation of this study is that it is based on only one case study. This makes the generalizability of the results to other systematic reviews of other interventions or other adverse effects difficult. Moreover, most of the trial reports with fractures came from the GSK register, and other pharmaceutical company reports may not provide as much detail.
It was also difficult to maintain consistency in the search strategy among different interfaces for the different databases to make fair comparisons. In addition, it was not possible to conduct any type of cost analysis of searching each source due to the complex pricing mechanisms used by database providers, which can be dependent on type of organization, size of network, number of concurrent users, and which provider the database is purchased from.
CONCLUSIONS
This case study demonstrates the potential value of searching several sources to identify adverse effects data. In this instance, a combination of searching the GlaxoSmithKline (GSK) Web site, Science Citation Index (SCI), EMBASE, BIOSIS Previews, British Library Direct, Medscape DrugInfo, handsearching, reference checking, AHFS First, and Thomson Reuters Integrity or Conference Papers Index (CPI) retrieved all the relevant references.
The case study here also demonstrates the failure of a broad search strategy with numerous synonyms, text words, and indexing terms to identify all the relevant references available on each database. This was mainly due to a lack of fracture terms in title, abstract or indexing/keywords of bibliographic records. This again emphasizes the need for authors of systematic reviews of adverse effects to search a wide range of sources and authors of studies to ensure adverse effects terms appear in the title or abstracts.
SUPPLEMENTARY MATERIAL
Supplementary Table 1
Supplementary Table 2
Supplementary Table 3
Supplementary Table 4
Supplementary Table 5
Supplementary Table 6
CONTACT INFORMATION
Su Golder, MSc, MRC Fellow in Health Services Research, University of York, CRD, Heslington, York, United Kingdom
Yoon K. Loke, MD, Senior Lecturer in Clinical Pharmacology, University of East Anglia, Norwich Research Park, Norwich, United Kingdom
CONFLICT OF INTEREST
Su Golder's institute has received a PhD Fellowship grant for her work from the Medical Research Council. The other author reports having no potential conflicts of interest.
