A research gap is a topic or area for which missing or inadequate information limits the ability of reviewers to reach a conclusion on a given question. A research need, on the other hand, is a research gap which needs to be filled to help decision makers. Identifying and prioritizing these needs is a key component in developing a research agenda to meet the needs of patients, clinicians, payers, and other decision makers in health care. The National Institutes of Health (NIH) invests over $31.2 billion annually in medical research in the United States (US) (14). Efforts must be made to ensure that investments are directed toward research that addresses research needs.
Guidelines development committees are in an ideal position to identify research gaps. These committees often include content experts for relevant specialties (clinicians, nurses, therapists, etc.), methodologists (experts in assessing evidence and/or developing guidelines, and health economists and statisticians if appropriate), representatives of potential stakeholders, and patients/consumers (21). In addition to different perspectives, these groups also encompass a wide array of skills including expertise in clinical domains, epidemiology, biostatistics, and health services research (Reference Shekelle, Woolf, Eccles and Grimshaw18). In developing evidence-based guidelines, the guidelines committee considers the available evidence, identifying its strengths and limitations. These discussions are a valuable source of research gaps.
There has been limited research on the identification of gaps. We identified several studies, in specific disease areas, that described formal methods for identifying gaps from systematic reviews (Reference Chou, Ballantyne, Fanciullo, Fine and Miaskowski5;Reference de Vet, Kroese, Scholten and Bouter7;Reference Johnson, Proctor and Farquhar12;Reference Shepherd, Briggs and Payne19). Clarke et al. (Reference Clarke, Clarke and Clarke6) examined systematic reviews to assess whether and how research recommendations were made. Of the more than 2,500 Cochrane reviews evaluated, less than one in five specified the population (P), intervention (I), and comparison (C) of interest.
Research on research needs has focused on methods for prioritization and presentation (Reference Brown, Brunnhuber and Chalkidou3;Reference Chalkidou, Whicher, Kary and Tunis4;11;Reference Noorani, Husereau, Boudreau and Skidmore15). For example, Brown et al. (Reference Brown, Brunnhuber and Chalkidou3) suggested the use of the EPICOT+ framework to characterize and present research recommendations, including the existing (background) evidence (E), population (P), intervention (I), comparison (C), outcomes (O), and time stamp (T). We found no systematic effort to identify research gaps from guidelines. In this study, we pilot-tested a method to identify and classify research gaps from evidence-based guidelines.
METHODS
We reviewed all evidence-based guidelines documents published by the Cystic Fibrosis Foundation (CF Foundation). We abstracted recommendation statements and their grades, as assigned using the grading system of the United States Preventive Services Task Force (USPSTF) (Reference Sawaya, Guirguis-Blake, LeFevre, Harris and Petitti16). Consensus recommendation statements made after a priori acknowledgment of the insufficiency of the evidence and registry data-based recommendation statements were also abstracted, but were excluded from further analyses as we did not consider these evidence-based statements.
Our research team has experience in conducting multiple systematic reviews to inform evidence-based guidelines. One author (I.J.S.) completed the identification, classification, and assessment of explicitness of research gaps. This was reviewed by the other two authors (K.A.R. and N.A.M.).
Identification of Research Gaps
We used the following criteria to identify research gaps:
Insufficient evidence. (a) Topics for which recommendations were not made: These were topics for which statements such as “The CF Foundation concludes that there is insufficient evidence. . . . . .”; “the CF Foundation has insufficient evidence to make a recommendation regarding. . . . .”; or USPSTF grade I (insufficient) statements were made. (b) Topics for which consensus recommendations were made after evidence consideration: For these topics, the committee considered the evidence (the strength of evidence and certainty of benefit were reported), but consensus recommendation statements were made; and
Needing further research. (c) Topics or questions specifically identified by the guidelines document authors as needing further research: These were identified by the use of terms like “needing further research,” “further research needed,” or “future research directions.” These suggestions for further research were typically found at the end of guidelines documents in the Discussion or Future Directions sections.
Classification of Research Gaps
By Type of Management Issue. We classified research gaps by type of management question: (a) Comparative effectiveness: The Agency for Healthcare Research and Quality (AHRQ) defines comparative effectiveness research as a type of healthcare research that compares the results of one approach for managing a disease to the results of other approaches (1). We classified a research gap as a question of comparative effectiveness if different interventions would need to be compared to address the research gap. (b) Long-term effects/chronic interventions: We classified research gaps as addressing long-term effects/chronic interventions if they addressed the benefit of chronic treatment modalities, usually lasting for months to years. (c) Implementation/integration into practice: We classified research gaps as addressing implementation or integration into practice if they were directed toward real-world management issues like order of treatment entities, individualization of treatments, or assessment of responsiveness in patients with certain co-morbidities. (d) Clinical assessment: We classified research gaps as addressing clinical assessment if they related to either specific diagnostic tests or the benefit of specific clinical tests in the ongoing assessment of patients with cystic fibrosis (CF).
If appropriate, research gaps were classified as more than one type of management issue.
By Specificity of Target Population. We classified research gaps by the target population addressed: general CF population, specific CF populations (e.g., asymptomatic, with nutritional/growth deficits, with pulmonary exacerbations).
By Age of Target Population. We also classified research gaps by the age of the target population using the categories: infants and children <6 years of age only; older children (≥6 to <18 years of age), adolescents (≥12 to <18 years of age), and adults (≥18 years of age) only; adults (≥18 years of age) only; any age group.
Explicitness of Research Gaps
We assessed research gaps in terms of whether sufficient information was provided to use in developing research questions. We considered whether research gaps explicitly characterized the population (P), intervention (I), comparison (C), and outcomes (O).
RESULTS
We identified five evidence-based guidelines documents from the CF Foundation. These documents, published between 2007 and 2009, addressed a variety of topics related to the management of CF, including the use of chronic medications for maintenance of pulmonary health (Reference Flume, O'Sullivan and Robinson9); nutritional management (Reference Stallings, Stark, Robinson, Feranchak and Quinton20); the use of airway clearance therapies (Reference Flume, Robinson and O'Sullivan10); the treatment of pulmonary exacerbations (Reference Flume, Mogayzel and Robinson8); and the care of infants diagnosed with CF (Reference Borowitz, Robinson and Rosenfeld2).
The five guidelines documents included forty-eight original overall questions (mean, 9.6; median, 8; range, 4–21). Several of these questions included sub-questions such as those related to differing comparisons, differing doses of the same intervention, or differing frequencies of the same intervention. When these sub-questions were taken into account, there were eighty questions (mean, 16; median, 16; range, 4–32). Overall, eighty-nine recommendation statements were made in the guidelines documents (mean, 17.8, median; 14; range, 4–44). Of these eighty-nine recommendation statements, fourteen did not qualify for our analysis (ten consensus statements made after a priori acknowledgment of the insufficiency of the evidence and four registry data-based recommendation statements). Thus, our analysis included seventy-five recommendation statements.
Twenty-six (34.7 percent) of these recommendations were based on available evidence (USPSTF grades A, B, C, and D), while forty-nine (65.3 percent) did not have sufficient evidence (see Table 1). Where there was not sufficient evidence, the authors of the guidelines documents either did not make recommendations due to insufficient evidence (USPSTF grade I statements, n = 22) or made consensus statements (n = 27). Table 1 also provides the number of each grade of evidence-based statements (using the USPSTF grading system) made in each guidelines document. Of the seventy-five recommendation statements, only two (2.7 percent) were considered to be based on sufficient enough evidence to be grade A statements.
Table 1. Summary of Number of Recommendation Statements and Research Gaps in Evidence-Based Guidelines Produced by the CF Foundation
*USPSTF = United States Preventive Services Task Force
Overall, we identified sixty-two research gaps from the five guidelines documents (mean, 12.4; median, 9; range, 5–32). Of these, forty-nine research gaps (79.0 percent) were either grade I statements or consensus statements made due to insufficient evidence. The other thirteen research gaps (21.0 percent) were topics explicitly identified by guidelines document authors as needing further research (see Table 1).
We classified forty-four (71.0 percent) of the identified research gaps as issues of comparative effectiveness, twenty-nine (46.8 percent) as issues of long-term/chronic interventions, eight (12.9 percent) as issues of implementation/integration into practice, and six (9.7 percent) as issues of clinical assessment (see Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2011016). Note that where appropriate, research gaps were classified as addressing more than one type of management issue. One research gap (“How should we evaluate new methods of airway clearance?”) could not be classified as addressing any of our categories as that this was not a management issue but an issue of study design.
Forty (64.5 percent) research gaps related to the general CF population. Thirty-four (54.8 percent) research gaps were issues specifically related to infants and children less than 6 years of age. Of these, thirty-three research gaps related specifically to infants less than 2 years of age. Twenty (32.3 percent) research gaps were not specific to any particular age group (see Supplementary Table 1).
Among the sixty-two research gaps identified, there was sufficient information for us to characterize the relevant population (P), intervention (I), comparison (C), and outcomes (O) for 53 (85.5 percent), 49 (79.0 percent), 4 (6.5 percent), and 10 (16.1 percent) research gaps respectively. All components (P, I, C, and O) of a research question were available for only one of the identified research gaps.
On average, the process of reviewing guidelines documents, abstracting recommendation statements and research gaps, classifying research gaps, characterizing research gaps using the PICO framework where possible, and assessing the explicitness of research gaps took 3 days per guidelines document. Barriers to the process included difficulty in assessing research gaps identified as “needing further research” and identifying relevant comparison groups for research gaps when not explicitly stated.
DISCUSSION
We developed a systematic process of using evidence-based guidelines to identify research gaps within a specific disease area. We identified sixty-two research gaps from five evidence-based guidelines. Only approximately 20 percent of the gaps were specifically called out by guideline committees as research gaps. Relying on the future research sections would miss the majority of the research gaps identified through the guideline development process.
Our results also suggest that there is a need for guideline document authors to be more explicit in characterizing their recommendations for further research. For only one of the sixty-two research gaps were we able to clearly determine each component (P, I, C, and O) of a well-designed research question. This result is similar to that obtained by Clarke et al. (Reference Clarke, Clarke and Clarke6) in their assessment of future research recommendations from over 2,500 Cochrane reviews. They found that, although 82 percent of Cochrane reviews included a suggestion about a specific intervention, only 17 percent specified intervention, type of participant and outcome measure, and 12 percent did not specify any of these three components of a research question. Guidelines committees should be as specific as possible in stating what research is needed and why (21). The potential value of recommending further research is lost if research reports, systematic reviews, or guidelines provide general research recommendations (Reference Brown, Brunnhuber and Chalkidou3).
Encouraging guidelines committees to use standard frameworks, such as EPICOT+, may be helpful (Reference Brown, Brunnhuber and Chalkidou3). Recommending research in a standard format can help in two main ways. First, it can help those conducting research to identify research gaps in a clear and explicit manner. Second, it can help funding agencies prioritize research needs, by laying out important details of the relevant population, follow-up time, ideal study design, and disease burden each of which have important funding implications.
Because we assessed only those evidence-based guidelines documents published by the CF Foundation, we were restricted to a small sample size of five documents. As evidence evolves, we expect these guidelines documents to be updated. We envision our work to be ongoing to incorporate these updates as well as new evidence-based guidelines. Although we have used CF as an example, and limited our sample to US-based guidelines, the method we developed to identify research gaps could be applied in other settings, including other countries and for different diseases. Our method is inexpensive, transparent, and reproducible.
Our pilot completed the first step in a systematic process to identify research needs from evidence-based guidelines. We identified research gaps and characterized them using a standard framework (PICO) where possible. The next step is the translation of these research gaps into researchable questions. We plan to develop a database to include these research gaps, along with the recommendation statements from the guidelines. Our goal is to produce an up-to-date, comprehensive, and searchable database of clinical practice recommendation statements and research gaps on the management of CF.
A database similar to the one we have planned is DUETs (Database of Uncertainties about the Effects of Treatments) (13). DUETs includes questions about the effects of treatments which have not yet been answered by reliable up-to-date systematic reviews. Our database will not be limited to questions of treatment, and will include, for example, clinical assessments (including diagnostic tests and monitoring). Contributions to DUETs are currently limited to those within the United Kingdom drawing on three sources to identify uncertainties: contributed questions from patients, carers, and providers; future research recommendations found in systematic reviews and clinical practice guidelines; and ongoing systematic reviews and primary studies. In contrast with DUETs, our proposed database will also include research gaps identified as recommendation statements with insufficient evidence. Finally, our plan is to have a database that will include both research gaps and recommendation statements.
The third step is the prioritization of the research gaps, including identifying which of these are research needs. Noorani et al. (Reference Noorani, Husereau, Boudreau and Skidmore15), in related work examining priority setting for health technology assessment topics, identified twelve frameworks and fifty-nine unique criteria. Others have described a variety of processes for setting research priorities (Reference Chalkidou, Whicher, Kary and Tunis4;11). The variability in the specific criteria and processes reflects the different stakeholders and circumstances of priority setting. Our goal is to systematically identify and characterize research gaps to facilitate the development of research agendas by various stakeholders.
A final step is to establish a feedback mechanism to disseminate the prioritized research needs to relevant stakeholders, including funding agencies. This step necessitates not only a clear presentation of these needs, but the identification of a receptive stakeholder. For organizations, such as the CF Foundation, that are involved with guidelines development and with the development of research agendas, this link may be more direct. It is also a link that demonstrates the additional value and impact of guidelines on research. We have presented to the CF Foundation the research gaps identified in this pilot study, but a more formal feedback mechanism has not yet been established. The establishment of an efficient feedback mechanism may require a culture change from research as investigator-driven to a greater emphasis on collaborative and purposeful allocation of scarce research resources (Reference Scott, Moga, Harstall and Magnan17).
CONCLUSIONS
Guidelines are a rich source of research gaps. However, guidelines have not been widely or systematically used to identify research gaps. Using cystic fibrosis as an example, we developed and pilot tested a method to tap into this valuable source. This transparent method to systematically identify and classify research gaps from guidelines requires further testing in other disease areas. The potential impact is the full use of the guideline committee's expertise to not only guide practice but to also help inform research agendas.
CONTACT INFORMATION
Karen A. Robinson, PhD (krobin@jhmi.edu), Assistant Professor of Medicine, Ian J. Saldanha, MBBS, MPH (isaldan1@jhmi.edu), Senior Research Program Coordinator, Naomi A. McKoy, MS (nmckoy1@jhmi.edu), Research Assistant, Department of Medicine, Johns Hopkins University, 1830 E. Monument Street, Baltimore, Maryland 21287
CONFLICT OF INTEREST
Karen Robinson and Ian Saldanha inform their institution has received a grant from the CF Foundation for this work. Naomi McKoy reports she has no potential conflicts of interest.
