Healthcare decision makers aim to make timely, evidence-informed policy or practice decisions using the best possible research knowledge on a given subject. Evidence producers aspire to generate the highest quality research to inform these clinical or policy questions posed across a variety of disciplines using validated methods and transparent, rigorous process. The gold standard for evidence synthesis is the systematic review, which uses transparent, repeatable, and rigorous methods to comprehensively answer research questions (1–Reference Morton, Levit, Berg and Eden3). It takes time to thoroughly search, extract, appraise, and summarize evidence, and production times for systematic reviews reflect this (median 61 weeks, interquartile range [IQR] 33–87) (Reference Schünemann and Moja4–Reference Pai, McCulloch and Gorman6).
Healthcare decision makers (also called knowledge-users) often require information in a shorter time period and may be compelled to take action before an evidence review is complete, diminishing the impact of the systematic review (Reference Hailey7). Rapid reviews have been proposed as an alternative synthesis approach when there is an emergent need for evidence review in a shortened timeframe. Rapid reviews are characterized as a type of accelerated systematic review with no commonly accepted or validated methodology or definition (Reference Ganann, Ciliska and Thomas8–Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10). They have become increasingly popular as evidence producers struggle to balance the requirements of the decision-making process with the need for high quality, inclusive, evidence summaries.
There is no shortage of research on rapid reviews, yet a paucity of knowledge in specific content areas exists. Terminology used to report or publish rapid reviews is relatively diverse, and research attempting to characterize rapid review methodologies has demonstrated a heterogeneous continuum of methods, often similarly grouped by nomenclature or label only (Reference Ganann, Ciliska and Thomas8;Reference Harker and Kleijnen9;Reference Hartling, Guise and Kato11–Reference Hartling, Chisholm, Thomson and Dryden13). Additionally, producers and users of rapid reviews may have different views on what constitutes a rapid review. Although standards exist for the methodological conduct and reporting of systematic reviews and other types of evidence syntheses (Reference Moher, Liberati, Tetzlaff and Altman14;Reference Shamseer, Moher and Clarke15), there is no comparable gauge for rapid reviews. It is generally accepted that rapid reviews use tailored methods to expedite accepted evidence synthesis processes with the goal of shortening delivery time frames or adapting to limited production resources. There is currently no accepted definition that explicitly characterizes what makes a rapid review distinct from other forms of evidence synthesis. Despite this, interest in this type of product is growing an indication that healthcare decision makers are both interested in, and consuming this type of report (Reference Tricco, Antony and Zarin16;Reference Kamel and Mann17).
The absence of an accepted rapid review definition is well-documented (Reference Ganann, Ciliska and Thomas8–Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10;Reference Khangura, Polisena, Clifford, Farrah and Kamel18). They have been conceptually described as a concept, method, or an approach but we have no meaningful way to answer the question “what is a rapid review?”. While reduced production timeframes are considered a key feature of rapid reviews, a wide array of times to completion from 1 week to over 9 months or more has been described by previous work. Attempts to clarify or characterize common traits have been carried out by scoping the literature for methods descriptions or guidance for conduct or through the examination of minute details in samples of rapid reviews with a goal of summarizing key features (Reference Ganann, Ciliska and Thomas8;Reference Harker and Kleijnen9;Reference Watt, Cameron and Sturm19;Reference Watt, Cameron and Sturm20). Although a better understanding of these products was achieved through these efforts, the scarcity of research in this area combined with the inconsistency of methods, approach, and reporting for rapid reviews has resulted in the consistent message that we currently have no established definition for a rapid review.
Frequent descriptors can be extracted from previous work and there has been some progression in the descriptions of rapid reviews in the literature since 2008. Watt et al. (Reference Watt, Cameron and Sturm19) first described rapid reviews as a “concept” that meets the imperatives of both knowledge users and evidence producers. While this provides a thoughtful way to group similar, timely evidence reviews, it offers no context to further delineate this grouping or distinguish it from other approaches. Ganann et al. (Reference Ganann, Ciliska and Thomas8) captured fundamental variables of timeliness and knowledge synthesis in their description of rapid reviews and linked rapid reviews to the tailoring of full systematic reviews by summarizing them as a method to “streamline traditional systematic review methods to synthesize evidence in a shorter timeframe”.
Khangura et al. (Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10) furthered this notion by characterizing rapid reviews as having an aim to inform the emergent needs of decision makers in a healthcare setting. Abrami et al. (Reference Abrami, Borokhovski and Bernard21) use the term “brief review” to describe what they consider to be rapid reviews as they assert that both time and scope should be emphasized in the definition. The only other descriptive feature associated with rapid review comes with the acknowledgement that rapid review is, in fact, a set of disparate approaches. This deviates from the initial thinking that a single approach could be captured in a definition and distinguished from other evidence synthesis methods. Harker and Kleijnen (Reference Harker and Kleijnen9) and others describe rapid reviews as a “spectrum” of approaches not having a single definition while Polisena et al. (Reference Polisena, Garritty, Kamel, Stevens and Abou-Setta22) describe rapid reviews definitions as fluid or flexible. However, there is little clarity on this concept, and insufficient detail on whether this range of reviews can be further delineated (Reference Harker and Kleijnen9;Reference Khangura, Polisena, Clifford, Farrah and Kamel18;Reference Garrity23).
A 2014 survey of fifty-seven International Network of Agencies for Health Technology Assessment (INAHTA) member agencies produced a definition of rapid reviews in context with other HTA products:
“A Rapid Review will always describe the characteristics and current use of the technology; and, evaluate safety and effectiveness issues. (It will) often conduct a review of only high level evidence or of recent evidence and may restrict the literature search to one or two databases. (It will) optionally critically appraise the quality of the evidence base; or, provide information on costs/financial impact.” (Reference Merlin, Tamblyn and Ellery24)
This somewhat strict definition accounts for the variation that often accompanies descriptions of rapid reviews through its categorization of traits using “always”, “often” and “optionally”; However, the description confines the types of research questions (safety and effectiveness) that rapid reviews should answer, which in turn limits the universal applicability of the definition outside of HTA agencies (and arguably within some of their own member agencies). It also limits the extent to which a rapid review may be considered “systematic” by leaving discretionary options for single database literature searching and the omission of critical appraisal. It may be beneficial to explore whether rapid reviews can be more comprehensively defined using the themes that appear in the rapid review literature, such as the requirement to inform decision or policy makers, time to completion, conduct or approach as compared to other evidence synthesis products, the requirement for a structured literature review, or possibly even by the limited depth or focus that a rapid review may adopt. Further research is needed to capture a definition of rapid reviews that is more generalizable to a wider representation of evidence producers and knowledge users.
A clear, transparent, and sufficiently detailed definition of rapid reviews is necessary to solidify the placement of rapid reviews on the evidence continuum and to better inform healthcare decision makers about the value and appropriateness of these products. A modified Delphi process was used to achieve consensus on the defining characteristics of rapid reviews, and to demonstrate how these products are distinct from other forms of evidence synthesis based on the opinion of experts in the field of evidence synthesis and rapid reviews.
METHODS
We used a three-round modified Delphi method to discover what key stakeholders consider being the defining characteristics of rapid reviews (Figure 1). Ethical approval for this study was obtained through the Ottawa Hospital Research Ethics Board.
Figure 1. Flow chart of the modified Delphi process.
The Delphi Process
The Delphi method is commonly used to obtain input from a group of experts in a structured, iterative manner (Reference Boulkedid, Abdoul, Loustau, Sibony and Alberti25;Reference Lilja, Laakso and Palomki26). Central to the method is the use of controlled feedback between successive survey rounds and statistical “group response” measured graphically or numerically through measures of central tendency, dispersion, and frequency distributions (Reference Okoli and Pawlowski27). The classical Delphi method has several predefined rounds, or may continue until a prespecified level of consensus is achieved. The number of rounds required to reach stable consensus, and even what constitutes final consensus is a topic for debate in the literature (Reference Heiko28); however, a minimum of two rounds is required and the number of rounds used should be selected pragmatically (Reference Thangaratinam and Shakila29). Hsu et al. (Reference Hsu and Sandford30) suggest that there is no agreement on what the criterion for consensus should be in a Delphi study, but propose that the use of a percentage alone is inadequate. They advise the use of reliable alternatives in addition to any designated cutoff for agreement, such as “stability of subject responses in successive iterations” (Reference Hsu and Sandford30). Another beneficial characteristic of the Delphi method is the ability to provide anonymity to study participants through the use of online or email survey rounds.
The Delphi method used for this study is comparable to the classic Delphi in terms of process (i.e., a series of rounds with selected experts) and objective (i.e., to arrive at consensus); however, the modified online approach varied from the traditional use of in-person meetings in the first and subsequent survey rounds. In addition, the first round of the Delphi process traditionally involves an open-ended questionnaire aiming to have an expert panel map or outline key concepts on the topic under study (Reference Heiko28). We modified the process in this study by mapping key concepts and themes related to rapid reviews before the first round survey using a comprehensive review of the literature. This adaptation was carried out to maximize the response rate in the first round, and to provide a solid foundation for this study based on previous research on rapid reviews.
We prespecified three survey rounds and targeted 70 percent for consensus on each statement describing rapid review characteristics along with the absence of significant issues noted in the free-text comments, as suggested in Hasson et al. (Reference Hasson, Keeney and McKenna31) and Hsu et al. (Reference Hsu and Sandford30). Two rounds were considered adequate to perform item generation, drafting of characteristics descriptors and obtaining initial agreement from the expert panel, with the goal of finalizing key rapid review characteristics in the third and final round. The number of rounds was limited intentionally so that experts were informed about the length of commitment to expect, limiting potential fatigue among participants and maximizing response rate.
Setting
Online survey (first round) and e-mail communication (second and third rounds).
Participants
We invited a purposive sample of 100 experts to participate in this study through an email invitation. The sample of experts was not limited to a minimum or maximum number. The 100 experts initially contacted was the natural result of the sampling process we used for this work, and not an arbitrary cap set by the investigators. The experts were identified through a detailed search of relevant literature, conference Web sites, key membership associations, and by referral of content specialists. Experts targeted for participation were: (i) subject-matter experts or experienced researchers with knowledge of a variety of evidence synthesis methods and practical experience with rapid reviews, (ii) authors with publications relevant to rapid reviews, and (iii) delegates presenting pertinent work at recent conferences or symposia (Cochrane Colloquium, Health Technology Assessment international (HTAi), CADTH Symposium, Cochrane Canada Symposium).
Standardized email invitations contained a letter asking experts to participate, a description of study objectives, methods, expected time commitment, consent information, and a link to the FluidSurveys (32) Web site hosting the first round of the Delphi study. Consenting participants were asked to refer individuals they considered suitable based on the objectives of the study, personal knowledge of rapid reviews, and content expertise. Twenty additional experts were identified through participant snowball referrals.
Consensus
An a priori level of 70 percent agreement was set for the agreement on the defining characteristics, which had to be accompanied by stability in the free-text comments (Reference Hsu and Sandford30).
First Round: Questionnaire Development and Generating Themes on Rapid Reviews. The first round questionnaire was developed and designed explicitly for this study using FluidSurveys as the platform (33). Content topics and themes for the survey instrument were selected from a review of the literature on rapid reviews. Comprehensive literature searches were conducted with the assistance of an experienced medical information specialist knowledgeable in evidence synthesis and rapid reviews. Search strategies were peer reviewed and used both controlled vocabulary (e.g., National Library of Medicine's MeSH terms) and keywords. On October 25 and 31, 2011 we searched PubMed, MEDLINE, EMBASE, the Cochrane library, York Centre for Reviews and Dissemination (CRD) Database of Abstracts of Reviews of Effects (DARE), NHS Economic Evaluation Database (EED) and HTA, Web of Science, National Library of Medicine Gateway, and CINAHL (EBSCOHost).
Search updates were carried out monthly in PubMed until December 31, 2014. A thorough search of conference and meeting abstracts was also carried out. The search strategy is available from the corresponding author by request. Relevant publications were screened and selected by a single study author (S.K.). We considered research on rapid review methodology, applications, impact and utility along with related research on timely or accelerated evidence syntheses, tailoring research methodologies and the needs of policy makers as they pertain to the use of research evidence. Following the literature review, relevant concepts, themes, issues, and knowledge gaps were translated into a Web-based English-language questionnaire.
The first round survey consisted of seventeen broad questions addressing a variety of rapid review themes aimed at soliciting opinion and open comments from the expert panel based on their knowledge and experience with rapid reviews. The goal of this survey was to solicit rankings and free-text comments on a comprehensive set of questions with the goal of eventually prioritizing those most characteristic of a rapid review. For our purposes, defining characteristics are defined as “one of a number of essential features by which a rapid reviews can be recognized” (Reference McKenna34). Specific themes identified from the literature review and addressed in the survey were: (i) time lines for conduct, and how rapid reviews and traditional systematic reviews may differ; (ii) research questions and protocols; (iii) tailoring or omitting specific methodological components of a systematic review for a rapid review approach; (iv) reporting; (v) appropriateness of rapid reviews in decision making; and (vi) requirements for guidelines on conduct.
Nine individuals representative of a cross-section of the expert population piloted the questionnaire and checked the accompanying instructions on two separate occasions. The questionnaire was completed and checked for face and content validity, clarity, feasibility, and to ensure that the practical elements of design were appropriate. Feedback from pilot testing on two separate occasions resulted in refinement of the survey tool and modifications to the wording of questions and instructions. The first round of the Web-based survey was circulated by email invitation in July 2013 and consenting experts were asked to respond to the survey within four weeks (Supplementary Table 1).
Panelists were asked to rank the importance of specific methodological and reporting components on a 5-point Likert scale where “1” was “very unimportant” and “5” was “very important.” Subsequent questions addressed a variety of topics using categorical input options. A comment box was placed after each question, and at the end of the survey, to encourage free-text comments and to ensure all participants had sufficient opportunity to make their views known on the different themes explored, or to provide context for the answers provided.
A brief demographic questionnaire was also administered to gain a better understanding of perceived expertise in systematic reviews and rapids review, main institutional affiliation (by category) and job specialization. Fluid Surveys collected information on the respondents’ country of origin by default. To maximize response rate, the initial survey invitation was followed by two reminder emails separated by 2-week intervals. Individuals who had not responded to the survey and not declined consent were contacted by email one final time before closing the survey, and given a 5-day grace period to submit their surveys.
We analyzed responses from each round separately. Response and completion rates, along with expert attrition, were monitored and recorded throughout the survey rounds.
Second Round: Development of Draft Defining Characteristic Statements. Following the first round, survey responses were analyzed quantitatively. We used frequency distributions to review categorical data and to summarize questions with ordinal response scales. Text-responses from open-ended questions on the amount of time required to conduct a systematic review or a rapid review were standardized from days, weeks, months, and years into weeks for analysis. An informal, yet structured content analysis of all free-text comments was carried out, specifically capturing word frequency and emergent themes using NVivo software (35). To condense data for the second round, major themes and concepts and those identified by Likert scale as being “important” or “very important” by 70 percent or more of the expert panel were extracted and selected for inclusion in draft characteristic statements and presentation in the second round survey. Themes were compiled and presented in the form of draft language statements proposed as the defining characteristics of rapid reviews along with a text summary from the first round that was agreed upon by all study authors.
A second round was administered in May 2014 to experts who responded to the initial round. The second round survey endeavored the panel to agree or disagree with the proposed draft language statements on rapid review characteristics using an agree/disagree format and invited them to comment or suggest wording changes using free text. A summary of responses and comments from the first round was also distributed to each expert to provide context for how the language statements were assembled. Respondents were asked to return the survey document by email within 4 weeks, and reminder emails were set-up at 10-day intervals.
Third Round: Characteristic Statement Refinement. A third round was administered in August 2014 to experts who responded to both the first and second rounds. Responses from the second round were used to edit the provisional statements on the defining characteristics of rapid reviews based on expert agreement and free-text comments received. Draft statements circulated in the previous round were restated for reference, and the refined statements were proposed for consideration. Experts received an individualized report showing the pooled anonymous group response (percent who agreed) to each statement on rapid reviews and their own individual response from the second round (agree/disagree). Experts were given the opportunity to reconsider their own response in context of the group response using a standardized form with room for free-text comments.
RESULTS
Expert Panel Profile
A total of sixty-six experts consented to participate in the study (response rate 55 percent). Forty-two experts were sent the survey and all email reminders with no response. Three experts refused the initial survey invitation, six viewed the online questionnaire but did not complete any question, two contacted the first author directly to decline participation citing limited time to commit to the study and a single person responded that although they had experience with systematic reviews and rapid reviews, they believed that their current level of knowledge in the area was inadequate to be considered an expert for the purposes of our study.
A profile of the respondents is provided in Table 1. Experts who consented to participate were predominantly from North America (76 percent) and the United Kingdom (15 percent).
Table 1. Profile of the Expert Panel (n = 66)
a Participants were asked to rate their perceived expertise in systematic and rapid reviews on a scale from 0 to 100, where 0 represented ‘beginner’ and 100 represented ‘expert’.
b Other response was ‘research consultancy’.
c Other included: health economist (n = 1), librarian (n = 1), public health physician (n = 1).
Respondents self-rated their expertise in systematic and rapid reviews on a scale from 0 (beginner) to 100 (expert). Experts rated themselves as having a moderately high to high level of expertise in both systematic reviews (median, 79.0) and rapid reviews (median, 75.5). Individuals on the expert panel have not been identified to maintain the anonymity agreed upon as a condition of participation.
First Round
Sixty-two of sixty-six expert panelists completed the first round survey in full (93.9 percent completion rate). Experts were diligent in qualifying the answers they provided through free text comments and were helpful in highlighting additional relevant topics not addressed in the first round questionnaire. They asserted that time to completion for both systematic and rapid reviews may be confounded by how the start (e.g., after topic refinement completed or following protocol approval) and finish (e.g., when the report is completed, when it is provided to the end-user, when published) of the review is defined. Similarly, the resources available (e.g., in terms of available reviewers, budget or other organizational restrictions such as a limited number of reviews produced per year) can influence the time to completion for a review, and may be impacted by whether reviewers are paid or volunteer.
The complexity or scope of the review was also noted as a potential impact on the time to completion. The variation and requirement for multiple rapid review approaches was also addressed, as experts asserted that variation in method is potentially tied to variation in time to completion. Certain respondents thought “rapid review” was too broad of a term to capture all approaches under a single “catch-all” phrase. Comments also asserted that accelerating time to completion to meet a decision making need was the only justification for modifying systematic review methods.
Across a variety of different questions (e.g., optimal types of research questions, reporting, inclusion of internal or external peer-review), reviewers commented that the “best” answer was the one that ultimately meets the needs of the decision maker. There was a suggestion that some types of research questions may be too burdensome to answer using a rapid review approach (e.g., safety or harms, development of guidelines).
Experts were resolved that rapid reviews must have a protocol (or at minimum a document that outlines objectives and approach before the review) and that bibliographic databases must be searched when conducting a rapid review (e.g., versus a simple Web search using Google Scholar). Most comments received reflected a desire to maintain as much rigor as time will permit, and that tailoring of processes should be review-dependent and may impact the ability to claim that the review is systematic on completion.
Comments across all themes reflected the need to tailor approach, methods and reporting to the requirements of the decision maker, in close consultation, when there is an emergent or urgent decision required.
Second Round
In the second round, forty-four experts provided their agreement on eleven draft characteristic statements circulated (response rate = 66.7 percent; completion rate = 95.5 percent). Remaining experts did not provide feedback following three email reminders.
Supplementary Table 2 lists the eleven draft statements reflecting conduct, reporting, and other key characteristics associated with rapid reviews in the opinion of the expert panel. Consensus and general stability in free-text comments was achieved for nine of the eleven statements circulated. While experts statistically agreed (>70 percent) on the fundamental concepts in each of the nine statements, many comments suggested minor edits to the wording or language used. Full consensus was not achieved for two statements addressing risk of bias assessment and the idea of rapid reviews being a unique methodology. Comments reiterate the experts’ distinctly opposing views in these areas rather than disagreement with the wording or language.
Third Round
In the final round, twenty-six remaining experts rated each of the eleven draft statements a second time. Before circulation, edits were made to the phrasing of each statement based on feedback from the second round (Response rate 59.1 percent, Completion rate 100 percent). Consensus was achieved for ten of the eleven characteristics in the final round including eight which had expert consensus of 90 percent or higher (Supplementary Table 2). No statistically significant increases in statement agreement between rounds two and three were observed.
Experts agreed on nearly all of the key characteristics generated for rapid reviews, but contrasting opinions persisted on the subject of risk of bias assessment and these were reflected in the free-text comments received (Supplementary Table 2, statement 11).
Consensus was reached on: (i) the utility and appropriateness of rapid reviews in answering a variety of research questions; (ii) the requirement for a rapid review protocol outlining approach, PICO, objectives and scope at minimum; and (iii) variable reporting length. These characteristics predominantly reflected the expert opinion that the approach to rapid reviews must be based on the particular needs of knowledge-users, the audience for which the rapid review is intended. Based on the experience of the expert panel, these must be balanced with the feasibility of answering those needs in a particular timeline, while maximizing transparency and reproducibility. Additional statements with a high level of expert agreement represented a variety of concepts on rapid reviews. Experts agreed that, in their opinion, the time to completion of a rapid review is significantly less than that of a systematic review (median 9 weeks; IQR 5.0 to 13.0 compared with median 35 weeks; IQR 21.7 to 52.1) and rapid reviews can be defined by the time taken to complete an evidence synthesis related to a defined research question(s) using the most systematic or rigorous methods as time allows.
Experts did not agree that rapid reviews are a unique knowledge synthesis method, but established that further research is required to better understand the impact of various workflow adaptations on the validity of the final rapid review product. A general summary of rapid review characteristics is provided in Table 2.
Table 2. Summary of Rapid Review Defining Characteristics, Based on Expert Opinion
DISCUSSION
The aim of this project was to achieve a pragmatic, operational definition for rapid reviews that is useful to both evidence producers and users, and allows for differentiation of this approach (or spectrum of approaches) from other synthesis methods. Sixty-six experts from six different countries agreed to participate in this modified online Delphi study. An analysis of their profile shows that they are an experienced group of researchers with a high level of expertise and familiarity with rapid reviews. The panel of experts included prominent researchers in the area of evidence synthesis who had an adequate and appropriate background to contribute to the development of a definition for rapid reviews.
This is the first study to our knowledge to explore this topic using the iterative input of experts to work toward a consensus definition. We used a three-round modified Delphi approach to query the anonymous expert panel on what they considered to be the key characteristics of rapid reviews, or the essential features by which a rapid review can be recognized. Experts reached consensus on ten of eleven final statements considered. Within this collection of statements, full consensus (100 percent agreement) was achieved for three statements and a very high level of consensus (≥90 percent agreement) was attained for eight statements. None of the statements showed a significant change in agreement (p ≥ 0.05) between the second and third survey rounds, which is suggestive of stability in the expert sentiment (Reference Abrami, Borokhovski and Bernard21). Results demonstrate that rapid reviews can be differentiated from systematic reviews or other evidence synthesis products using these key defining characteristics, although there may be overlap among the common traits.
In our study, experts agreed that a comprehensive definition of rapid reviews should consider multiple domains, including: (i) timeliness in relation to full systematic reviews; (ii) an aim to meet the specific requirements of a decision maker; (iii) tailoring of the explicit and reproducible methods conventionally used to conduct systematic reviews to expedite the process; (iv) flexibility of approach, process adapt and turnaround time; (v) the existence of risk to the overall validity of the process and findings that must be considered; and (vi) a goal to use the most rigorous and systematic methods that time will allow; and (vii) the essential requirement of transparency of process and completeness of reporting in all output from the rapid review.
Experts also agreed on several essential methodological components that rapid reviews should aim to comply with, including the requirement for a protocol (describing at minimum scope, population, intervention, comparator and outcome, and the approach), a search of bibliographic databases, an internal or external peer review process, although no further detail on the content came out of this work. Although consensus was achieved on ten of eleven characteristic statements, experts did not agree on the statement describing assessment of the risk of bias within or across studies included in a rapid review, and attempts to clarify or revise statements focused on this domain were not successful (Supplementary Table 2, statement 11).
This work follows that of other research groups who have attempted to extract or compile a definition of rapid reviews through different means, including comprehensive investigations of methods papers or sample rapid reviews (Reference Ganann, Ciliska and Thomas8–Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10;Reference Khangura, Polisena, Clifford, Farrah and Kamel18;Reference Hailey, Corabian, Harstall and Schneider36). A fulsome analysis of forty-six full rapid reviews and three extractable summaries of rapid reviews by Harker and Kleijnen (Reference Harker and Kleijnen9) did not elucidate a clear or final definition for what actually constitutes a rapid review. Similar to previous empirical studies of rapid reviews by Watt et al. (Reference Watt, Cameron and Sturm19), Ganann et al. (Reference Ganann, Ciliska and Thomas8), Harker and Kleijnen (Reference Harker and Kleijnen9), and more recently Polisena et al. (Reference Polisena, Garritty, Kamel, Stevens and Abou-Setta22), and Hartling et al. (Reference Hartling, Guise and Kato11), this consensus exercise found that experts agreed that, in their experience, rapid reviews can be defined by the time variables and referenced relatively against the timeline for a systematic review.
Experts largely held the opinion that rapid reviews should be conducted in less than 3 months and acknowledged the need for variable approaches, which in turn, requires acknowledgement of variability in times to completion (Supplementary Table 2, statement 9). The narrow spread of the IQR suggested for rapid review turnaround time indicates less variability, however, when compared with the IQR suggested for systematic reviews. The wide spread of the IQR for systematic review timeframes reiterates concerns previously stated in the literature about the inconsistency of timelines required to produce these types of structured syntheses. It is notable that the IQRs do not overlap across products, although there is considerable overlap in the first quartile. The makes it difficult to make anything more than a relative comparison between the rapid and systematic reviews. Experts believed strongly that the time component should be qualified with the aim to be as systematic or rigorous as the time negotiated with the end-user will permit when answered a research question.
The expert panel agreed (Supplementary Table 2, statement 1, 100 percent agreement) that rapid reviews are useful for answering efficacy or effectiveness research questions, a thought mirrored in previous articles by Merlin et al. (Reference Merlin, Tamblyn and Ellery24) and Hailey et al. (Reference Hailey, Corabian, Harstall and Schneider36). However, results also showed that rapid reviews are potentially useful for any type of research question. There has been suggestion in the literature that extra caution should be heeded where legal implications must be considered (Reference Polisena, Garritty, Kamel, Stevens and Abou-Setta22) and that rapid reviews of existing guidelines are appropriate, but should not be considered to supply evidence to guidelines development processes (Reference Coates37) although experts in this study did not consistently raise those issues.
The suitability of a rapid review to answer a particular research question should be mutually determined by both the evidence producer and the knowledge user commissioning the report, and suitability should be balanced with the user's context for the question and the ability of the evidence producer to adequately answer the question within the requested timeframe. This close consultation with the commissioner of the rapid review is important and has been suggested in previous reports (Reference Ganann, Ciliska and Thomas8;Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10;Reference Khangura, Polisena, Clifford, Farrah and Kamel18). Ganann et al. (Reference Ganann, Ciliska and Thomas8) highlighted the requirement for a well-formulated question tied to clear and definite context from the decision maker, which was echoed in many of the free-text comments from experts in this study. It can be inferred from this exercise that there may be research questions for which a rapid review is not appropriate or feasible; however, experts were clear that decisions relating to the appropriateness of rapid review approaches for a particular research question should be considered individually based on the needs of the commissioning end-user.
There is no clear delineation as to when a systematic review becomes “unsystematic,” but if what makes a review systematic is the use of an explicit protocol that can be checked for accuracy, then indirectly, any review not adhering to a protocol is not systematic (Reference Sandelowski38). Without a protocol, we are unable to check whether a review did what it said it would do, nor can we truly assess whether arbitrary decisions were made that may bias findings (Reference Shamseer and Moher39). Statement 2 (Supplementary Table 2, 100 percent agreement) captures the expert panel's clear requirement for a protocol to be written before a rapid review. There was no consensus as to what should be included in a rapid review protocol, if specific guidelines for rapid review protocols are desired or needed or whether registration of these protocols is necessary or feasible.
Comments from experts reflected a dual purpose for the rapid review protocol. First a protocol should be a type of contract between the evidence producer and the knowledge user which defines objectives and scope in a mutually agreeable manner with a goal of expectation setting. Second, the key objectives of the systematic review process must be upheld, including transparency and repeatability of procedure. In work analyzing over one hundred rapid reviews, Tricco et al. (Reference Tricco, Antony and Zarin16) state that protocols were not mentioned in 98 percent of included samples. Polisena et al. (Reference Polisena, Garritty, Kamel, Stevens and Abou-Setta22) explored the methods of HTA agency rapid review programs and conversely report that 96.6 percent of HTA agencies incorporate protocol development into their processes, yet only 51.7 percent of programs prepare protocols in conjunction with the requestor. It is clear that improved reporting, registration and/or publication of rapid review protocols, coupled with better coordination between researchers and decision makers is necessary going forward to meet this important provision. Further research is required to determine if an extension of current reporting guidelines to rapid reviews would increase compliance (i.e., PRISMA-P) and whether there is value in this exercise.
Issues of methodological process were raised by the expert committee. A comprehensive search for all published and unpublished literature is a key feature of the systematic review process (40). The expert panel mirrored these sentiments and achieved consensus that not only is structured searching of electronic databases a key defining characteristic of rapid reviews, it is one that axiomatically differentiates rapid reviews from less rigid evidence products. This characteristic fails to distinguish rapid reviews from other rigorous evidence synthesis products (e.g., systematic reviews). The systematic ideal for eligibility assessment, data extraction and quality appraisal is two independent reviewers (1). Ganann et al. (Reference Ganann, Ciliska and Thomas8), Harker and Kleijnen (Reference Harker and Kleijnen9), Khangura et al. (Reference Khangura, Konnyu, Cushman, Grimshaw and Moher10), Polisena et al. (Reference Polisena, Garritty, Kamel, Stevens and Abou-Setta22), and Hartling et al. (Reference Hartling, Chisholm, Thomson and Dryden13) all report that rapid reviews commonly revert to a single reviewer for one or all of these steps to expedite a review or adapt to resource constraints.
The expert panel agreed that a flexible number of reviewers at any stage of the review process is a defining characteristic of rapid reviews (Supplementary Table 2, statement 6, 92 percent agreement). When time permits, it is ideal to have a second review author check all or a portion of the work for errors, agreement or inter-rater reliability, but the panel acknowledged that this may be an acceptable place to concede if it is not feasible when negotiating delivery time. A large number of comments by the expert panel reflected requirement desire to accurately translate the potential risks limiting reviewer numbers to the end-user.
This study was unable to achieve consensus on quality assessment and whether use, omission or modification of these processes are a key defining feature of rapid reviews. Attempts to clarify the draft wording between the second and third rounds reduced agreement from 66.7 percent to 44 percent, and likewise, no fundamental topic agreement was noted in the expert comments. As per the Cochrane Handbook (1), assessment of the validity of the findings of the included studies is a key characteristic of a systematic review. Experts held opposing views on this subject, which, at the most basic level, speaks to the very incongruous opinions on what even constitutes a rapid review altogether (Reference Ganann, Ciliska and Thomas8).
In free-text comments, experts who supported the statement the inclusion of risk of bias assessment in rapid reviews posited that without an evaluation of the validity of the included studies, a review cannot be considered “systematic.” Similarly, they suggest that without a risk of bias assessment, it is impossible to communicate to the end user the risk that they will over or underestimate the true effect under study. This may be reflective of the views of experts with proficiency in reviewing health technology interventions, although it was not possible to confirm this. Experts opposing the risk of bias statement were also very firm in their stance.
The general view was that there was no time to conduct risk of bias assessments when producing timely evidence reviews and that their knowledge users preferred to obtain the available evidence quickly and then carry out their own assessments for risk of bias, if warranted. These opposing views on risk of bias assessment may be symptomatic of the knowledge gaps that persist in the current literature, such as whether rapid review should always aim to be systematic, or should it simply map evidence for the end user, possibly preceding a systematic review? Furthermore, should the broader umbrella concept of “rapid review” include both of these approaches? The results of this study support use of the most rigorous (or systematic) methods possible within the allowable timeline, and allow for a variety of approaches to rapid review. This allows for tailoring of review methods, including the risk of bias assessment, if agreed on by both the research producer and the knowledge user or recipient.
Experts broadly agreed that rapid reviews make use of a spectrum of approaches to complete an evidence synthesis related to one or more defined research questions using the most systematic and rigorous methods as a limited time frame will allow. There was also general agreement that rapid reviews tailor the clear, reproducible and well-documented methods typically used when carrying out a systematic review in some manner to expedite the review process. Additionally, experts established that any tailoring of process or approach should be done in a clear and transparent manner in consultation with the commissioning decision maker, making clear the potential risk to the validity of the rapid review that has yet to be quantified in the literature (Reference Hartling, Guise and Kato11;Reference Tricco, Antony and Zarin16). The panel directive for an open and transparent communication and consultation process between the researcher and the end-user is considered essential to the rapid review approach(es) and is central to the eventual uptake and impact of the rapid review product.
While this study reports expert-delineated characteristics of rapid reviews, no single definitive statement was produced to explicitly differentiate rapid reviews from other evidence synthesis methods. We were unable to eliminate overlap with systematic reviews or scoping reviews, and possibly, it was unfitting to attempt this feat. This has been noted in a related study by S. Kelly, where participants unanimously agreed that it is inappropriate to endeavor to find a single, unique methodology the broad array of report types that fall under the term “rapid review” (unpublished data, 2015). This finding serves to emphasize the requirement to move toward an alternative means of defining rapid reviews. A classification scheme may offer a more pragmatic way to capture the spectrum of approaches used to expedite evidence synthesis processes, and may better meet the diverse requirements of both evidence producers and users who rely on rapid reviews.
STRENGTHS AND LIMITATIONS
The strength of this study is the use of a validated methodology aimed at explicitly reaching consensus on a definition of rapid reviews. We chose this technique over other consensus methods because of suitability for the topic; the systematic and rigorous nature of the method; low cost; capacity to overcome geographic limitations; and ability to ensure an anonymous, equal voice to all of the expert panelists. The bulk of respondents on the expert panel were from Canada, with smaller numbers from the United Kingdom and United States. The small number of experts from outside these countries limits the extent to which we can say that our findings are representative of the views of a truly international set of experts. The manner of purposive sampling, combined with snowball referrals from experts contacted may have contributed to the potential imbalance in geographical representation on the panel (Reference Campbell and Stanley41). Despite several attempt to mitigate attrition, response rate for the second and third rounds of this survey were moderate and this may influence the validity of the final results (Reference Hasson, Keeney and McKenna31). We had limited resources to measure or identify how attrition may have biased our results, if at all. Nonetheless, we consider our findings to be a novel and informative addition to the existing knowledge base on rapid reviews.
It is important to note that the consensus definition of rapid reviews resulting from this work may not be the best definition or even a correct definition. It is, rather, a definition based on defining characteristics for rapid reviews that a group of experts considered appropriate. Other research producers or knowledge users may have different perspectives from this expert panel on what constitutes a rapid review, and may have much different thresholds on what is considered “timely” evidence. It is important to note that the definition produced here does little to eliminate overlap with systematic review definitions, although we may not be able to differentiate these two closely related products in any further detail without exploring additional categorization of rapid reviews into specific “sub-types” of approach (e.g., evidence map, rapid review, rapid systematic review).
The selection of experts for this study included individuals who have published relevant literature or contributed to existing research on rapid reviews. The process for selecting potential experts and using a snowball technique to obtain participant referrals may have resulted in a grouping of experts with similar backgrounds, views, or experience in this research. This may limit the representativeness of the participants. This may have also influenced the number of participants with similar ideologies continuing on to later rounds of this study and potentially increased the likelihood of consensus achieved within the group.
CONCLUSION
This modified Delphi study achieved consensus on ten statements describing characteristics of rapid reviews based on the opinion of a panel of experts knowledgeable in evidence synthesis methods and rapid reviews. In addition, it highlighted areas of disagreement, particularly where quality assessment was considered. According to the expert panel, rapid reviews aim to meet the requirements and timelines of a decision maker commissioning the review and should be conducted in less time than a systematic review. They use a variety of approaches to accelerate the evidence synthesis process, tailor the methods conventionally used to carry out a systematic review and use the most rigorous methods that the delivery time frame will allow.
Our findings emphasize the role of the decision maker in the rapid review process and stress the importance of transparent reporting. Future research should explore the applicability of these findings across the various approaches, or types, of rapid reviews and to test effective use in practice. These results are an essential contribution to our understanding of rapid reviews and help to further define the functional concept of rapid review that has been used to-date. Our findings may assist evidence producers and knowledge users with conveying information about rapid reviews, their approaches and results to colleagues, review commissioners, and other health care decision makers.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: http://dx.doi.org/10.1017/S0266462316000489
Supplementary Table 2: http://dx.doi.org/10.1017/S0266462316000489
CONFLICTS OF INTEREST
T.C. is an employee of the Canadian Agency for Drugs and Technologies in Health (CADTH). S.K. and D.M. declare that they have no competing interests.
