Rationale for Peer Review of Searches at the Outset of Evidence Syntheses

As noted above, peer review of searches should be undertaken prior to the searches being run. The aim of peer review at this stage of the evidence synthesis is to detect and remedy any errors or omissions prior to the study selection phase, in order to reduce not only the risk of missing relevant studies but also the risk of identifying unnecessarily large numbers of irrelevant records.

Searches, however, are not always conducted or reported to a high standard (Reference Mullins, DeLuca, Crepaz and Lyles3;Reference Layton4). As the search strategy is the cornerstone underpinning any well-conducted evidence synthesis, its quality could affect the results of the final analysis. A study published in 2006 by Sampson and McGowan (Reference Sampson and McGowan5) found that errors in search strategies were common, the principal mistakes being spelling errors, missed spelling variants, truncation errors, logical operator errors, use of wrong line numbers, missed or incorrect use of medical subject heading index terms (e.g., MeSH), and the search strategy not being tailored for use in other databases. A more recent study published in 2018 by Franco et al. (Reference Franco, Garrote, Escobar Liquitay and Vietto6) assessed a random sample of seventy Cochrane systematic reviews of interventions published in 2015, evaluating the design and reporting of their search strategies using the recommendations from the then current Cochrane Handbook for Systematic Reviews of Interventions (2011 version) (Reference Lefebvre, Manheimer, Glanville, Higgins and Green7), the then current Methodological Expectations of Cochrane Intervention Reviews [MECIR] standards (2013 version) (Reference Chandler, Churchill, Higgins, Lasserson and Tovey8), the then current version of the Peer Review of Electronic Search Strategies (PRESS) evidence-based guideline (2009 version) (Reference Sampson, McGowan, Cogo, Grimshaw, Moher and Lefebvre9), and the then current PRESS Evidence-Based Checklist (2010 version) (Reference McGowan, Sampson and Lefebvre10). They found problems in the design of the search strategies in 73 percent of the reviews (95% CI, 60–84%) and 53 percent of these contained problems (95% CI, 38–69%) that could limit both the sensitivity and precision of the search strategies. More recently, a study published in 2019 by Salvador-Olivan et al. (Reference Salvador-Olivan, Marco-Cuenca and Arquero-Aviles11) found that 92.7 percent of their sample of 137 included systematic reviews, published in January 2018, contained some type of error in the MEDLINE/PubMed search strategy and that 78.1 percent of these errors affected recall/sensitivity.

It must be recognized, however, that not all librarians and information specialists are based in teams and so may be unable to call upon colleagues to peer review their searches. Under such circumstances, the use of the PRESSforum Peer Review (https://pressforumpr.wordpress.com/) is recommended. This forum has been established to enable librarians and information specialists to submit their searches for peer review by a fellow librarian or information specialist, on a reciprocal basis.

Rationale for Peer Review of Searches Immediately Prior to Publication

As outlined above, peer review of searches should be undertaken prior to the searches being run. While this is increasingly being recognized as best practice, it is certainly not yet universal. There remains, therefore, an important role for peer review of searches immediately prior to publication, in common with standard peer review practice within the scientific community.

Peer review of searches immediately prior to publication, however, is rarely undertaken by suitably qualified and experienced librarians or information specialists. A recent survey of librarians and information specialists by Grossetta Nardini et al. (Reference Grossetta Nardini, Batten, Funaro, Garcia-Milian, Nyhan and Spak12) found that they are rarely approached to participate in the peer review of systematic reviews or their search strategies at the prepublication stage (see below). The recent launch of the Librarian Peer Reviewer Database (https://sites.google.com/view/mlprdatabase/home/about-this-site), which serves to connect librarians who have expertise in searching for evidence syntheses with journal editors who need peer reviewers with expertise in this area, should go some way toward remedying this situation.

In April 2020, four of the major international library associations (the Canadian Health Libraries Association (CHLA/ABSC), the European Association for Health Information and Libraries (EAHIL), Health Libraries Australia (HLA-ALIA), and the U.S. Medical Library Association (MLA)) submitted a joint letter to the International Committee of Medical Journal Editors (ICMJE) urging journal editors to actively seek information specialists as peer reviewers for knowledge synthesis publications and to advocate for the recognition of their methodological expertise (Reference Iverson, Della Seta, Lefebvre, Ritchie, Traditi and Baliozian13).

Which Organizations Advocate Peer Review of Searches?

In the recently revised chapter entitled Searching for and selecting studies in the Cochrane Handbook for Systematic Reviews of Interventions (Reference Lefebvre, Glanville, Briscoe, Littlewood, Marshall, Metzendorf, Higgins, Thomas, Chandler, Cumpston, Li and Page1), it is strongly recommended that search strategies be peer reviewed. An evidence-based checklist such as the PRESS Evidence-Based Checklist should be used to assess which elements are important in peer review of electronic search strategies (Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre14;Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre15). It is not only that the checklist covers the technical accuracy of the strategy (line numbers, spellings, etc.) but also that the search strategy covers all relevant aspects of the research question and has interpreted the question appropriately. The names, credentials, and institutions of the peer reviewers of the search strategies should be noted in the review (with their permission) in the Acknowledgments section. This builds on Cochrane's experience in publishing systematic reviews of diagnostic accuracy studies, where peer review of the searches has been mandatory since inception at both the outset of the review and at the prepublication stage (16). The Cochrane Information Specialists’ Executive is actively encouraging a culture of peer review of searches for Cochrane intervention and other reviews, by Cochrane Information Specialists, at the outset of the review and has carried out a pilot project in this area (17). It should also be noted that, under Cochrane's new membership scheme, membership points are awarded for peer reviewing search strategies (https://www.cochrane.org/join-cochrane/contribution-types-and-membership-points).

The Centre for Reviews and Dissemination in the U.K., in its guidance entitled Systematic reviews: CRD's guidance for undertaking reviews in healthcare in the section entitled Appendix 2: example search strategy to identify studies from electronic databases, states: “If at all possible, the final search strategy should be peer reviewed to check for errors (spelling mistakes, incorrect use of operators or failure to include relevant MeSH) that could reduce the recall of papers” (18).

The European Network for Health Technology Assessment (EUnetHTA), in its guidance entitled Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. Methodological Guidelines, in the section entitled Summary and table with main recommendations, states that “search strategies should undergo peer reviewing to ensure high-quality search strategies” (19). They add in the section entitled Chapter 3: Comprehensive information retrieval. 3.1.6. Peer reviewing search strategies: “The peer review process using the [PRESS] checklist should be completed before the search strategy is run.”

The Institute of Medicine in the U.S., in its guidance entitled Finding what works in healthcare: standards for systematic reviews in the section entitled Recommended standards for the search process, states, “Use an independent Librarian or other Information Specialist to peer review the search strategy” and in the section entitled Planning the search: ensuring an accurate search, states: “The peer reviewer should be independent from the review team in order to provide an unbiased and scientifically rigorous review and should have expertise in information retrieval and systematic reviews. In addition, the peer review process should take place prior to the search process, rather than in conjunction with the peer review of the final report, because the search process will provide the data that are synthesized and analyzed in the systematic review” (20).

The Institute for Quality and Efficiency in Health Care (IQWiG) in Germany, in its guidance entitled IQWiG General Methods in the section entitled Quality assurance of search strategies, states: “Quality assurance with the PRESS checklist is initially a formal evaluation and is always performed before the conduct of searches” (21).

The National Institute of Health and Care Excellence (NICE) in the U.K., in its manual entitled Developing NICE guidelines: the manual. Process and methods in the section entitled Quality assurance, states: “For each search (including economic searches), the principal database search strategy should be quality assured by a second Information Specialist to maintain a consistently high standard for identifying the evidence. A checklist should be used to ensure clarity and consistency when quality assuring search strategies. An example is the PRESS 2015 Guideline Evidence-Based Checklist (Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre15). Each time the principal database strategy is adapted for use in another database, it is good practice for it to be peer reviewed by a second Information Specialist to ensure quality and consistency is maintained” (22).

The Preferred Reporting Items for Systematic reviews and Meta-Analyses—Extension for Searches (PRISMA-S Extension), in the section entitled Peer review, states, “Describe any search peer review process,” and in the Explanation section of that same section, states: “Authors should consider using Peer Review of Electronic Search Strategies (PRESS) Guideline statement, a practice guideline for literature search peer review outlining the major components important to review and the benefits of peer reviewing searches (Reference McGowan, Sampson, Salzwedel, Cogo, Foerster and Lefebvre15). Authors should strongly consider having the search strategy peer reviewed by an experienced searcher, Information Specialist or Librarian” (Reference Rethlefsen, Kirtley, Waffenschmidt, Ayala, Moher and Page23).

The PRISMA 2020 statement, in Box 2 Noteworthy changes to the PRISMA 2009 statement, states: “Modification of the ‘Search’ item to recommend authors present full search strategies for all databases, registers and websites searched, not just at least one database (see item #7).” This will facilitate more comprehensive peer reviewing of the search process at the prepublication stage. The PRISMA 2020 Explanation and Elaboration document, in the section entitled Search strategy Item 7. Present the full search strategies for all databases, registers, and websites, including any filters and limits used, states: “Explanation: Reporting the full details of all search strategies (such as the full, line by line search strategy as run in each database) should enhance the transparency of the systematic review, improve replicability, and enable a review to be more easily updated. The description of the search strategy development process might include details of … any processes used to validate or peer review the search strategies. Empirical evidence suggests that peer review of search strategies is associated with improvements to search strategies, leading to retrieval of additional relevant records (Reference Spry and Mierzwinski-Urban31). Further guidance and examples of reporting search strategies can be found in PRISMA-Search” (Reference Rethlefsen, Kirtley, Waffenschmidt, Ayala, Moher and Page23).

Finally, the U.S. Agency for Healthcare Research and Quality (AHRQ) in its Methods Guide for Effectiveness and Comparative Effectiveness Reviews, recommends peer review of searches for the updating of evidence syntheses. In the section of its guidance entitled General search strategies for updating CERs (Comparative Effectiveness Reviews), it states, “Investigators should also use the opportunity to review the search strategy and modify search terms, databases and other sources searched, if necessary, and have it peer-reviewed, if not previously done” (26).

How to Keep Up to Date with the Evidence Around Peer Review of Searches

The web resource Summarized Research in Information Retrieval for HTA (SuRe Info) (27) provides research-based information relating to the information retrieval aspects of producing HTAs and systematic reviews (Reference Isojärvi and Glanville28). SuRe Info seeks to help information specialists stay up to date with the latest developments in information retrieval by providing easy access to current methods research and thus support more research-based information retrieval practice. In the General search methods section of the Web Site, there is a chapter entitled: Peer reviewing search strategies (Reference Lefebvre and Duffy2).

Since January 2013, searches have been conducted to identify research about the peer review of search strategies in HTAs and systematic reviews in order to populate this chapter (see Supplementary Table 1). The search strategies combine relevant search terms comprising indexed keywords (e.g., Medical Subject Headings [MeSH]) and free text terms appearing in the titles and/or abstracts of bibliographic database records. Search terms were identified through discussion with the SuRe Info project team, by scanning background literature and “key articles” already known to the project team and by browsing database thesauri. The search strategies were structured using search terms for “peer review” in combination with search terms for “literature searching.” Search strategies were developed specifically for each database and the keywords adapted according to the configuration of each database. Searches were not limited by language, publication status (e.g., unpublished or published), or date of publication. The searches are updated on a 6-monthly basis, conducted every March and October.

The following databases are searched:

• • MEDLINE; MEDLINE In-Process Citations, Medline Daily Update, and Epub Ahead of Print (Ovid interface)

• • PubMed (NLM interface)

• • Embase (Ovid interface)

In addition, citation searches of prominent search strategy peer review research articles are undertaken alongside the biannual update searches. Separate key author searches are also conducted. The citation and author searches are conducted in Google Scholar, Science Citation Index (SCI), and PubMed.

Full details of the search strategies and citation searches are provided in Supplementary Table 1. References identified from the searches are downloaded into EndNote bibliographic management software for further assessment, removal of duplicates and screening. Search results are screened for potentially relevant articles that fulfil the SuRe Info inclusion and exclusion criteria (see Table 1) as published in the SuRe Info: authors’ manual (29) and consensus is reached between the SuRe Info peer review chapter authors (CL and SD).

Table 1. SuRe info project team. SuRe info: authors’ manual: HTAi, 2017: inclusion and exclusion criteria

What Does the Current Evidence Tell Us About Peer Review of Searches?

The Agency for Healthcare Research and Quality (AHRQ) conducted a study in 2012 assessing a project they had undertaken on the peer review of search strategies and found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (Reference Relevo and Paynter30). The time burden of the review process including peer review of search strategies was less than 2 hours.

In 2018, CADTH (Reference Spry and Mierzwinski-Urban31) conducted an internal investigation to see whether peer review of search strategies affected the number and quality of articles included in CADTH Rapid Response reports and found that both the number and quality of relevant articles retrieved were improved. The results of pre-peer-reviewed searches from a sample of 200 rapid reviews were compared with their corresponding post-peer-reviewed search results. The peer-reviewed strategies identified additional records in 75 percent of the searches investigated. Of these records, 4 percent were included in the rapid review reports. It should be noted that the search strategies were developed quickly for rapid reviews, and so were not highly sensitive. Had the searches been aiming for higher sensitivity, the differences above may have been more pronounced.

A prospective longitudinal assessment of a continuing education for systematic reviews workshop for librarians published in 2020 by Folb et al. (Reference Folb, Klem, Youk, Dahm, He and Ketchum32) measured various systematic review search practices before and after the workshop. The authors found that, before the workshop, only 9 percent of survey respondents had ever provided peer review for other librarians and 36 percent had sought peer review of their own strategies. At the follow-up stage of the survey, after the continuing education course, they found that 17 percent had provided peer review of other librarians’ search strategies and 48 percent had sought peer review of their own strategies.

As noted above, research published in 2019 by Grossetta Nardini et al. (Reference Grossetta Nardini, Batten, Funaro, Garcia-Milian, Nyhan and Spak12) investigating librarians as methodological peer reviewers for systematic reviews found that few librarians are being asked to review systematic review manuscripts immediately prior to publication.

Anecdotally, we have noticed increased reporting of both peer reviewing of search strategies and use of the PRESS Evidence-Based Checklist in published systematic reviews and HTAs but without any reporting of evidence of the effectiveness of their usage. Despite the studies referred to above, we have been unable to identify any studies assessing whether the PRESS Evidence-Based Checklist or other formal approaches to peer review affect the final quality of HTAs or systematic reviews.