Introduction
The intention of a systematic review (SR) is to identify, evaluate and synthesize as much relevant evidence as possible (within resource and time limits) relating to a focused research question. An SR is also documented in such a way as to be reproducible (Reference Akobeng1). This structured process involves following a predefined protocol, conducting a thorough search for applicable evidence and providing an explicit description of methods. These steps set an SR apart from other forms of literature review and it is one of the reasons an SR is considered to be the highest form of evidence (Reference Akobeng1–3). A transparent, well-reported SR allows readers to assess for themselves the utility and value of findings. To aid clear reporting, the Preferred Reporting of Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist was published in 2009 (Reference Moher, Liberati, Tetzlaff, Altman and The4). PRISMA 2009 consists of a list of twenty-seven items which authors of SRs and meta-analyses are expected to include. In 2015, the ROBIS (Risk Of Bias In Systematic reviews) tool was published. The ROBIS tool is a domain-based tool that uses signaling questions in each of four domains to identify aspects of the SR design where bias may be introduced. An overall judgment of risk of bias is made after all domains have been assessed (Reference Whiting, Davies, Savovic, Caldwell, Reeves and Shea5). Despite the ubiquity of PRISMA 2009 and its endorsement by many journals, including high-impact journals such as The Lancet and BMJ, the reporting of search methods for SRs (covered by no. 7 and no. 8 of the PRISMA 2009 checklist) has not met requirements. Additionally, quality in some reviews has been further impacted by searching a limited range of resources, applying inappropriate search limits, and no searching for gray literature or reference checking (Reference de Kock, Stirk, Noake, Duffy, Ross and Misso6). The aim of this research is to evaluate the methodological and reporting characteristics of search methods of SRs published in 2016 using the PRISMA 2009 checklist (Reference Moher, Liberati, Tetzlaff, Altman and The4) and the ROBIS tool (Reference Whiting, Savović, Higgins, Caldwell, Reeves and Shea7).
Methods
Inclusion/Exclusion Criteria
Using the random number generator in Excel, a sample of 505 (out of 18,129) SRs published in 2016 was taken from KSR Evidence (8), a database of SRs in health care. SRs were not restricted by topic, language or journal. The sample of SRs included narrative reviews and meta-analyses but did not include reviews on methodology or genome-wide studies.
Data Extraction
A data extraction sheet was developed in Excel. The sample was divided between the six authors who are experienced in undertaking and reporting search methods for SRs. The questions were first discussed and trialed to ensure authors’ consistency and then the following was extracted singly for each SR:
(1) Does the SR claim to be following PRISMA?
(2) Does the SR comply with no. 7 (Information sources) of PRISMA 2009?
(3) Does the SR comply with no. 8 (Search) of PRISMA 2009?
(4) Does the SR comply with both no. 7 and no. 8 of PRISMA 2009?
(5) Does the journal require PRISMA compliance for SRs in instructions to authors? (It was not possible to retrospectively determine precisely when a journal first started requesting PRISMA compliance in instructions to authors. However, as the PRISMA statement was published in 2009, it was assumed that this was unlikely to have been later than 2016.)
(6) What is the judgment for D2 (Identification and Selection of Studies) from the ROBIS tool?
Assessment of Reporting Compliance
To assess reporting compliance with PRISMA 2009, a binary score of yes or no was given for compliance with the following PRISMA items:
-
(7) Information sources: Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.
-
(8) Search: Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated (Reference Moher, Liberati, Tetzlaff, Altman and The4).
Based on the explanation and elaboration of PRISMA 2009 (Reference Liberati, Altman, Tetzlaff, Mulrow, Gøtzsche and Ioannidis9), the authors considered reproducibility to be fundamental as to whether the reporting of PRISMA 2009 no. 7 and no. 8 was compliant. For example, a description of information sources needed to include the database, the database provider, the database date spans and the searching date; the search strategy needed to be provided in sufficient details to indicate how terms were combined with Boolean logic and what fields were searched. A list of databases without associated hosts or dates was not considered compliant. A list of keywords with no explanation as to how they were combined was not accepted. Ambiguities were discussed within the team until consensus was reached.
Assessment of Methodological Quality
To assess methodological quality, D2 (Identification and Selection of Studies) of the ROBIS tool was applied. ROBIS is not a checklist, like PRISMA, but consists of a set of signaling questions to aid a reader to make an overall judgment about the risk of bias. D2 (Identification and Selection of Studies) asks a reader to make a judgment on the following:
2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms of the structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
2.5 Were efforts made to minimize error in selection of studies? (Reference Whiting, Davies, Savovic, Caldwell, Reeves and Shea5)
The response to each of the above questions is either yes (Y), probably yes (PY), probably no (PN), no information (NI) or no (N). Based on the responses, the reviewer then makes a judgment as to whether the risk of bias for D2 is low, high, or unclear.
For this project, the overall ROBIS risk of bias of the random sample, including the risk of bias assessment for D2, was undertaken by two independent reviewers separately as part of the development of the KSR Evidence database that includes a subset of ROBIS-assessed critical appraisals of SRs. The authors, however, evaluated the findings for D2 to confirm agreement with the reviewers’ judgments and, in a small number of cases, made changes to the D2 judgment after discussion with other members of the team.
Results
Preferred Reporting of Systematic Reviews and Meta-Analyses
One hundred and ninety-nine SRs (thirty-nine percent) were compliant in no. 7 of PRISMA 2009 and reported information sources correctly. One hundred and seventy-four (thirty-four percent) reported a repeatable strategy of at least one database search. In total, ninety-seven (nineteen percent) SRs reported information sources and a search strategy such that search methods could be reproduced (see Table 1).
Table 1. SR compliance with PRISMA and ROBIS items
PRISMA no. 7, description of information sources; PRISMA no. 8: repeatable search strategy; ROBIS D2, identification and selection of studies; SR, systematic review; PRISMA, Preferred Reporting of Systematic Reviews and Meta-Analyses; ROBIS, Risk Of Bias In Systematic reviews.
Journals that Mention PRISMA in Instructions to Authors
The random sample of 505 SRs was published in 387 different journals covering a range of topics and languages. One hundred and sixty-seven (forty-three percent) of these journals recommended that PRISMA 2009 be applied in the reporting of an SR. In total, 238 SRs were published in these 167 journals that recommend PRISMA compliance in instructions to authors. Ninety-seven (forty-one percent) SRs published in journals which recommend PRISMA were compliant in the reporting of information sources. Eighty-four (thirty-five percent) adhered in the reporting of a repeatable search strategy. Fifty-seven (twenty-four percent) SRs were compliant in both no. 7 and no. 8 of the PRISMA 2009 checklist. This means that 181 (seventy-six percent) SRs that did not comply with PRISMA no. 7 and no. 8 were published in journals that request PRISMA 2009 reporting.
SRs that Mention PRISMA Compliance in their Methods
Two hundred and thirty-eight (forty-seven percent) SRs published in the sample claimed in their methods to be reporting according to PRISMA 2009. One hundred and four (forty-four percent) SRs that made this claim were compliant with the description of information sources, and ninety-two (thirty-nine percent) were compliant with the reporting of a search strategy. In total, forty-eight (twenty percent) SRs that claimed to be PRISMA-complaint in their methods reported both no. 7 and no. 8 of PRISMA 2009 adequately, so that search methods could be repeated.
ROBIS Assessment and PRISMA Compliance
Ninety-nine (twenty percent) SRs in the sample were judged to be at a low risk of bias (see Table 1), 141 (twenty-eight percent) were at an unclear risk of bias, and 265 (fifty-two percent) were considered to be at a high risk of bias in D2 of ROBIS. Forty-nine SRs (ten percent of all) were judged to have a low risk of bias in D2 of ROBIS and also complied with no. 7 and no. 8 of PRISMA 2009.
Discussion
Our findings indicate that the request for PRISMA reporting in journal instructions to authors is no guarantee that an SR will be PRISMA-compliant. Additionally, some journals such as PLoS ONE and Systematic Reviews require a PRISMA checklist to be included with SR submissions. Anecdotally, we noticed that sometimes the PRISMA checklist for items no. 7 and no. 8 pointed to the PRISMA flowchart with no detailed information being provided about the resources searched or strategies applied, so the requirement of a PRISMA checklist was no guarantor that PRISMA reporting was being complied with. As our study included SRs from 167 different journals that listed PRISMA as a requirement and noncompliance was an issue in more than three quarters (eighty-four percent), we would argue that the noncompliance of no. 7 and no. 8 is not restricted to a few journals or subject areas. In a study looking at the endorsement of PRISMA and quality of SRs and meta-analyses published in nursing journals, Tam et al. (Reference Tam, Lo and Khalechelvam10) also found that the endorsement of PRISMA did not change the adherence levels to the PRISMA statement. Riado Minguez et al. (Reference Riado Minguez, Kowalski, Vallve Odena, Longin Pontzen, Jelicic Kadic and Jeric11) had similar findings and stated, “Endorsement of a certain checklist is not a guarantee that SRs published in those journals will actually comply with the checklist.” Explicit mention of PRISMA compliance within the text of an SR is also no surety that search methods will adhere to the reporting requirements of PRISMA. Panic et al. (Reference Panic, Leoncini, de Belvis, Ricciardi and Boccia12) had similar findings to ours and could not find a connection to the explicit mention of PRISMA adherence in the text to an increase in reporting quality. Leclercq et al. (Reference Leclercq, Beaudart, Ajamieh, Rabenda, Tirelli and Bruyere13), on the other hand, found that the mention of PRISMA did have a positive influence on reporting completeness of meta-analyses indexed in PsycINFO but also concluded that compliance generally was far from optimal and that authors should be strongly encouraged to read PRISMA before starting.
With regard to the reporting of a replicable search strategy, it was common for search terms to be reported, but it was not always clear what fields, if any fields, were searched, how search terms were combined, or if a mixture of free text and thesaurus terms had been used. This compares to other studies on search strategy replicability. Koffel and Rethlefsen (Reference Koffel and Rethlefsen14) noted that only twenty-two percent of articles in a study of SRs published in 2012 provided at least one reproducible strategy (Reference Koffel and Rethlefsen14), whereas Page et al. (Reference Page, Shamseer, Altman, Tetzlaff, Sampson and Tricco15) found that only thirty-four percent of non-Cochrane reviews published in 2014 included search strategies with full Boolean search logic. Page et al. also noted that search dates were often only partially reported with either start and end dates for one database being reported or only one end date being reported for all databases.
In comparison with other studies on PRISMA reporting compliance, our findings for no. 7 and no. 8 of the PRISMA 2009 checklist differ. Although we have not been able to find published research that looks exclusively at the reporting compliance of the search methods of PRISMA, we have found a number of papers that have looked at compliance of all the twenty-seven items of the PRISMA 2009 checklist. A comprehensive scoping review was undertaken by Page and Moher (Reference Page and Moher16) to evaluate the uptake and impact of the PRISMA statement and the various PRISMA extensions. Twenty-seven studies in Page and Moher's scoping review looked at PRISMA adherence in SRs published in 2010 and after, so later than the publication of PRISMA in 2009. However, they found eighty-four percent of SRs complied with the reporting of information sources (no. 7) and sixty-two percent complied with the reporting of the search strategy (no. 8). Another large study, Nawijn et al. (Reference Nawijn, Ham, Houwert, Groenwold, Hietbrink and Smeeing17) looked at 112 SRs published in 2015 and 2016 in emergency medicine and stated that eighty-six percent reported information sources in compliance with PRISMA 2009 and fifty-three percent reported a search strategy in compliance with PRISMA 2009.
Differences between our research findings and other studies may be due to other PRISMA compliance studies investigating specific topics rather than a broad range of health care reviews. Comparable studies also had varying publication dates or looked at a range of specific journals. However, it could also be that other PRISMA compliance studies were conducted by reviewers rather than information specialists. Information specialists with expertise in searching methods are likely to be stricter when considering if PRISMA items no. 7 and no. 8 have been adhered to and ensuring that searches are reproducible. In addition, we also feel that the guidance provided in the explanation and elaboration of PRISMA, Liberati et al. (Reference Liberati, Altman, Tetzlaff, Mulrow, Gøtzsche and Ioannidis9), was unclear and open to misinterpretation for PRISMA no. 7. Liberati et al provides an example that lists the database coverage dates for Medline as 1966–present, CancerLit as 1975–present, and Embase as 1980–present. Although a final search date is provided in the example, it would be difficult to exactly reproduce the original search as the database segment dates were not provided and “present” is not specific enough.
The percentage of SRs that were both well reported (PRISMA 2009) and well conducted (D2 of ROBIS) in this sample was small (ten percent). However, the crossover between well-reported SRs and well-conducted SRs was higher in that forty-nine out of the ninety-nine well-conducted SRs were also well reported. It needs to be noted that an SR that was less transparent so, for example, did not report a repeatable search strategy and did not indicate whether searching limits such as language or date limits had been applied would be judged by D2 as being an unclear risk of bias. In contrast, a well-reported SR could more easily be judged to be at a high risk of bias as a reader is made aware of shortfalls in the conduct of the search methods. As the amount of health care research published continues to increase, the importance of SRs that appraise and summarize research to answer clinically relevant questions becomes ever more critical. However, if an SR is not transparent or comprehensive, conclusions cannot be relied or trusted, and this only serves to increase research waste.
While undertaking this research, the first draft of PRISMA-S (Reference Rethlefsen, Ayala, Kirtley, Koffel and Waffenschmidt18) was made available for consultation. PRISMA-S is explicit in how information sources should be described, clearly demarcating the difference between the database and the database host, dates of coverage and the searching date. All search strategies, not just one database search, are to be provided and all supplementary searches, not just database searches, are expected to be described in detail. This includes citation searching, text analysis and contact with experts. PRISMA-S is emphatic that search methods need to be replicable. Although PRISMA-S requires SRs to report with more detail than they have done to date, it is also made clear to readers why each individual item needs to be reported.
Limitations
We employed binary scoring for the compliance of no. 7 and no. 8 of PRISMA 2009 which does not support partial recognition of reporting compliance. Although ambiguities were frequently raised and discussed, it would have been better to have had two people independently extracting information and all disagreements discussed until consensus was reached. It is also possible that some journals added PRISMA compliance to their instructions to authors subsequent to the publication date of our sample (2016). However, we think this unlikely as PRISMA was widely publicized and most journals will have added this information to their instructions to authors prior to 2016.
Strengths
Our study using a random sample of over 500 SRs is considerably larger than most samples in other PRISMA-compliance studies. We also looked at health care in general, rather than specific topics, so we believe our findings are reflective of all SRs published in 2016. To our knowledge, this is also the only study to look specifically at the search methods of SRs using a combination of PRISMA for reporting and ROBIS for the conduct of search methods.
Conclusion
Based on our study, ninety percent of SRs are failing to report search methods adequately and to conduct thorough, comprehensive searches for available evidence. Undervaluing the importance of search methods and their reporting in the SR process will ultimately diminish the value of SRs, as evidence is the bedrock on which an SR is built on. To avoid these failings in SR methodology, we think it is essential that peer reviewers and journal editors are clear about what an SR should entail and what it means to be PRISMA-compliant. SRs need to have searched widely for evidence (two databases as a minimum) with a comprehensive search strategy that entails free text terms and thesaurus terms (when possible) combined with Boolean logic. The reporting of this needs to be clear and transparent so that the search methods can be repeated if necessary. It should not be acceptable to publish a review that says methods are reported according to PRISMA without ensuring that this is the case. Likewise, if a journal requests in instructions to authors that SRs are reported according to PRISMA, they should ensure that this too is done.
