A clinical research program was formed with the purpose of developing a new technology platform for measuring cardiac biomarkers at the patient bedside. A scoping review was performed by the authors to generate a list of potential early biomarkers of acute myocardial infarction to inform the clinical primary research program. The aim of this study is to describe the methodological approach adopted in the scoping review and a preliminary validation of the methods applied. The scoping review team included information specialists, systematic reviewers and members of the clinical research program. A range of established early biomarkers of myocardial infarction were already known to the clinical team before the review, whereas it was anticipated that the scoping review may also identify evidence relating to novel early biomarkers. The objectives of the scoping review were, first, to identify and clarify the names of potential early biomarkers to generate the required list of biomarkers and, second, to produce evidence summaries for listed potential biomarkers as selected by the clinical program team. Achievement of the review goals therefore relied on defining as broad a list as possible of potential biomarkers, including those not previously identified at the beginning of the review as being potential candidates for study.
Comprehensive searches (further details available upon request) were conducted by an information specialist across relevant electronic databases to identify evidence relating to potential early biomarkers of acute myocardial infarction. Sensitive keyword strategies using freetext and thesaurus searching were developed. Members of the clinical research program were consulted closely in the selection of search terms. Terms for myocardial infarction and acute coronary syndrome were combined with terms relating to their detection and diagnosis. Search records were then imported into reference management software for sifting. The term “sifting” is used in this study to describe the screening of search records for relevance to a review.
Challenges to the successful achievement of the review outcomes were several-fold. First, because the purpose of the scoping review was to identify the names of the biomarkers, these were by necessity not available in advance. Therefore, names of biomarkers could not be used as keywords and the search strategy was consequently based on generic terms relating to the detection of myocardial infarction and acute coronary syndrome. However, the use of generic search terms resulted in a very large search yield of over 27,000 records, with a high proportion of irrelevant records prevalent among these search results. In the traditional approach to sifting each individual search record would be examined in turn by the systematic reviewer. The authors estimated that, because approximately 500 search records might screened for relevance per day in a traditional sifting process, over 54 working days would be required for a single reviewer to exhaustively sift over 27,000 records. Therefore, the exhaustive sifting of each individual record within the extremely large search yield was not feasible within the timescales available for the rapid scoping review. Second, the open review question and the dynamic nature of marker relevance meant that there were no a priori inclusion criteria available. However, the fact that the primary outcome of the review was a list of potential biomarkers meant that the approach applied would not be required to identify every occurrence of a marker term but to capture sufficient references to explore the topic until term saturation had occurred and the primary outcome list had been generated. Therefore, the attributes of this scoping review necessitated the consideration of an alternative, more efficient and time-saving approach in which the most relevant evidence could be identified and prioritized over less relevant evidence for sifting.
An iterative sifting approach was applied, based on methods developed by Paisley (Reference Paisley1). The aim of the research described in this study was to undertake a preliminary validation of this iterative approach to the sifting or screening for relevance of search yields in reviews.
METHODS
The iterative sifting approach applied in our scoping review was based on several stages. The methods used are described in detail in this section and are summarized in Figure 1. As previously stated, the primary outcome of the scoping review was the generation of a list of potential early biomarkers. Evidence overviews were then to be produced for listed biomarkers selected by the clinicians as being of further interest. Given that the review was a rapid scoping review aimed at informing the scope of primary research and not a systematic review of the effectiveness of potential biomarkers, evidence summaries were to be based on studies within the reference management database and not on additional searches of bibliographic databases.
Figure 1. Diagrammatic summary of iterative sifting process.
The initial stage required the development of a primary collection of potential biomarkers and related terms with which to subsequently interrogate the reference management database and retrieve further, relevant evidence with which to refine and develop the list. First, key “index” papers were identified in conjunction with the clinical program team. Second, the reference management database was searched using terms (e.g., systematic review$, review$, overview$, progress, meta-analys$) to identify review articles that would provide an overview of the topic and further inform the emerging collection of terms. Third, the reference management database was interrogated using terms relating to innovation (e.g., new, novel, future, candidate) to generate any terms relating to new potential biomarkers. These three steps identified an initial relevant subset of references. The subset was sifted at title and abstract level for potentially relevant papers. The papers considered relevant were screened to generate a list of relevant, increasingly specific keywords with which to further interrogate the reference management database (e.g., assay$, elevat$ or peak$, bedside$, diagnos$, accura$, sensitivit$, specifit$, myoglobin, troponin, point-of-care). Titles, abstracts, and, where available electronically, full papers were screened to begin to generate the initial list of potential biomarkers.
At stage two, author searches were undertaken of the reference management database according to prevalent authors in the field identified from previously retrieved records. Subsequently retrieved records were then used to identify further relevant keywords. The clinical program team were consulted on the developing list of keywords to allow the inclusion of additional keywords.
The reference management database was then searched using the terms and keywords identified through the author search and through the previous search iterations. Retrieved records were tagged with appropriate keyword(s) as used in each search. Retrieved references were sifted and potentially relevant abstracts or full papers available electronically were screened for further search terms and for new biomarker name terms. The reference management database was explored using potential new search keywords until the generation of new marker names and search keywords appeared to be saturated, with no new names or keywords emerging.
Of the remaining unprocessed search records contained within the reference management database, a randomly selected sample of approximately 10 percent (n = 1,100) was screened at abstract level to confirm that, for the purposes of the scoping review, keyword saturation appeared to have been reached.
The iterations undertaken thus far were performed to achieve the primary aim of the review; to generate a list of potential early biomarkers for further consideration by the clinical program team. Once saturation appeared to have been achieved, the reference management database was finally interrogated using the list of potential early biomarker names identified as a result of the iterative sifting approach outlined above. Refinement of the list was then necessary to ensure that biomarker terms included on the list were clinically relevant to the team. Abstracts identified in the reference management database for each potential biomarker term were scrutinized. For a biomarker to remain on the list, there had to appear to be evidence of sufficiently early clinical detection of the cardiac biomarker as defined by the clinical specialists.
The refined list of potential early biomarkers was forwarded to the clinical program team as the primary outcome of the review. Several potential early biomarkers were highlighted by the clinical team as being of further interest. For each of the potential early biomarkers selected by the clinical team for further scrutiny, high level overviews of evidence for each selected marker were produced. These summaries did not seek to be comprehensive reviews of all the available published literature for each marker, but rather represented an overview of the scope of the evidence available for each selected biomarker. For this reason, the summaries were based on evidence at abstract level within the reference management database and from electronically available full text articles rather than on further search iterations of electronic databases. These biomarker summaries were presented to the clinical team to further inform their research program.
Following completion of the scoping review, it was decided to undertake a preliminary validation of the iterative sifting approach described above. Validation would demonstrate whether any biomarker names of potential relevance and interest to the clinical study team had been missed as a consequence of the iterative sifting approach. The preliminary validation was based on conducting an exhaustive sift of the remaining unsifted search records to assess whether any additional potential early biomarkers could be obtained that were not identified by the original iterative sifting approach.
The remaining unsifted search records were subjected to an exhaustive sifting process in which each individual record was screened in turn for relevance to the scoping review. Abstracts and full text of papers identified as being of potential relevance as a result of this sifting were examined and the names of potential early biomarkers identified were checked against the original list of biomarkers already identified. The reference management database was searched using the unique biomarker names that had been generated by the validation process but not identified by the iterative sifting approach. Any relevant abstracts or full text papers were examined for details of any clinically relevant change in marker levels. Any remaining biomarker names following this process were forwarded to the clinical program team for comments as to whether they would have been judged worthy of interest and whether an evidence summary would have been requested.
RESULTS
The searches undertaken for the rapid scoping review retrieved 27,198 records. These records formed the collection in the reference management database and constituted the full set of records subjected to the staged, iterative sifting approach and the subsequent validation process.
Stage one of the sifting process identified “index” articles and reviews / primary studies containing terms relating to innovation and operational / identity terms relating to biomarkers. This stage identified a subset of 2,846 records. When title and abstract sifting had been conducted the subset was reduced to 437 papers, which were used to identify new keywords with which to interrogate further the reference management database and to generate a primary collection of keywords and potential biomarker terms.
Stage two of the sifting process consisted of an author search and a search of the reference management database using the new keywords and potential biomarker terms identified by the author search and by stage one searches. This process was repeated until no new search terms or potential biomarker names were being generated. When this point of saturation was reached 16,254 (i.e., 59.7 percent) of the original 27,198 records had been sifted.
The random sampling (n = 1,100) of an approximately 10 percent sample of the remaining unsifted records (n = 10,944) identified no further potential early biomarkers, supporting the conclusion that keyword and marker name saturation had been reached as a result of the iterative sifting process.
As stated previously, the primary outcome of this rapid scoping review was to generate a list of potential early biomarkers to inform further clinical study. A total of 242 potential biomarkers were identified and included in an unrefined list. Markers in the list for which no evidence of a detectable change in level could be identified within the abstracts present in the reference management database were excluded. Evidence was identified within the reference management database to support the detection of fifty-three potential biomarkers (for which data were available on the time of biomarker measurement).
Therefore, the list of potential biomarkers forwarded to the clinical primary research program team contained fifty-three potential biomarker names. The clinical team considered nineteen biomarker names to be of particular clinical interest. Evidence summaries were produced for each of the selected nineteen biomarkers.
The iterative sifting process used in this scoping review, including the saturation check, resulted in the sifting of 17,354 (i.e., 63.8 percent) of the 27,198 records held in the reference management database. The remaining 9,844 records were subjected to the subsequent validation process following the completion of the rapid scoping review. When title and abstract sifting had been conducted, 143 records were considered as being of potential relevance. Scrutiny of these records at full text resulted in the identification of an additional sixteen potential early biomarker names that had not been identified by the original iterative sifting process. Of these, twelve biomarker names were based on a single paper only and six biomarker names were based on evidence that preceded the year 2000 or earlier. The clinical program team concluded that none of the additional potential biomarker names were considered worthy of additional scrutiny.
DISCUSSION
The iterative sifting approach applied in this rapid scoping review was successful in efficiently achieving the primary outcome of the required list of potential early biomarkers within the review timescales available. The use of the iterative sifting method in this review left 9,844 records unsifted, which were then subjected to the validation process. Based on a rough estimate of 500 search records sifted per day, the review team estimates that approximately 19.6 working days (9,844/500) were saved using this process. Although leaving a portion of search records unscreened could be viewed as carrying a risk in terms of potential unidentified evidence, in our case the preliminary validation of the iterative sifting approach indicated that the use of this iterative method did not result in the omission of any additional evidence that would have been of further clinical interest. The primary outcome of the list of markers and high level summaries of selected biomarkers were presented as a final report to the clinical specialist team and were used to inform their clinical research program and subsequent systematic review work (Reference Goodacre, Thokala and Carroll2).
A literature search was performed (May 2012) (search terms available on request) in an attempt to identify other reports of alternative methods of search record sifting. The only relevant example that could be identified of novel approaches to screening was the text mining work described by Thomas et al. (Reference Thomas, McNaught and Ananiadou3). Our approach can be considered as having some parallels with electronic automated text mining methods, as discussed by Thomas et al. as having potential for use in systematic reviews, in that such methods also seek to facilitate and increase the efficiency of evidence selection in systematic reviews, with the key difference that our iterative sifting approach was manual in nature.
The iterative sifting process applied in this work warrants further development and standardization for potential application in other reviewing and decision problem scenarios. In our case, the aim was to inform the scoping and planning of primary research to be conducted by the clinical team. The purpose was not to undertake a comprehensive systematic review of the effectiveness of early biomarkers. As such, the sifting method provided an efficient means of managing the results of an extensive scoping search and of generating a clearer definition of the clinical problem of interest to the clinical team. The same method could be applied to systematic reviews, particularly of complex or ill-defined topics where it is difficult to capture the definition of the problem in a single, discrete search process in the early stages of the review. The keywords and terms generated in the iterative sifting process could be incorporated in further rounds of searching of electronic databases with the purpose of maximizing the comprehensiveness of the search process. This iterative method may also have relevance for the identification of evidence within a reference management database for use in network meta-analyses or in populating decision-analytic models for cost-effectiveness assessments. Iterative sifting may also be considered appropriate in a rapid systematic review context, as an efficient means of retrieving evidence of highest relevance in a timely manner for prioritized data extraction and subsequent fast-tracked handling in the review process. For example, in a systematic review of clinical effectiveness of an intervention, it is desirable to quickly identify the most relevant evidence, for example randomized controlled trials (RCTs) of the effectiveness of an intervention. Therefore, using search terms for the specific intervention or drug name, terms for the drug class and terms for the study type (e.g., RCT, then perhaps other study types according to the appropriate hierarchy of evidence) could allow the prioritization of key clinical effectiveness evidence. In this way, a systematic reviewer could progress with data extraction, while a second reviewer completed sifting of the remaining, less relevant search records.
Despite the described iterative approach, the review team still sifted a proportion of search records that could be considered to be large. Further refinement of this method may elucidate whether it would be possible to increase the efficiency of this approach, yielding the same result but with a reduction in the sifted proportion of results and/or fewer steps applied in the process. It should also be noted that double sifting by two independent reviewers was not performed in this scoping review and subsequent validation, which might have acted as a potential source of bias. An additional stage could have been incorporated into the review process, had time permitted in this rapid scoping review, in that the review team might have taken the nineteen potential early biomarker terms as selected by the clinical team and used these to undertake a new round of comprehensive electronic database searches to further inform the research program.
CONCLUSIONS
An iterative sifting approach successfully increased the efficiency of a rapid scoping review. A preliminary validation of this approach indicated that no additional relevant evidence was omitted as a result of adopting this method. Following further refinement and standardization, manual iterative sifting may be of potential value in the rapid selection of evidence for reviews and other decision problems.
CONFLICTS OF INTEREST
The authors stated no conflicts of interest.
