Decision-analytic modeling is being used increasingly in the evaluation of health-care technologies. In 2001, an analysis of technology assessments used to support the National Institute for Clinical Excellence (NICE) appraisal process showed that 78 percent of assessments used some form of modeling approach (21). Concerns have been expressed about the quality of the evidence used in models and the lack of guidance in this area (15). The accuracy of the results of a model will be limited by the accuracy of the data it incorporates (7;9;13), which in turn will depend on the sources and methods used to identify the data (19).
Although methods for identifying the evidence relating to effects data are well established (6;18) and often evidence-based (8;29), and efficient methods of searching for economic studies are evolving (17), as far as we are aware, there are no generally accepted or tested methods to collect data to populate decision-analytic models (15;19). Although existing good practice guidelines emphasize the importance of incorporating available and appropriate information (1;9;10;13;24;26–28), they offer no guidance on how to identify evidence for the model (19). However, some authors (25;28) have made explicit reference to the use of a systematic approach to identifying data and others have indicated that good modeling practice should incorporate the best available evidence from all possible sources to limit the potential for bias (7;11;12;15).
The identification of data to populate decision-analytic models may require a very different approach from that used to identify effects data or economic evaluations (5;15). Yet only four of the twenty-seven technology assessments examined by Paisley (21) described any additional searches to populate the economic model. Decision models typically include a range of types of information, such as treatment effects, baseline event rates (which may relate to the natural history of the condition), quality of life effects, health state values (or utilities), and resource use and unit costs. One source of information can rarely provide data for all these parameters. For example, systematic reviews of randomized controlled trials (RCTs) will almost certainly be sought for treatment effect parameters, as these study types should provide estimates that are least susceptible to bias. However, RCTs and systematic reviews will generally not, on their own, be an adequate source of data for a decision model (7;10;12;19), because RCTs tend to have a short follow-up, may be undertaken in locations or with patient groups that are unrepresentative of routine practice, and are unlikely to measure resource use (12;19). Therefore, focusing on information from trials to populate all the parameters in a model may be inappropriate and information from research using other study designs may need to be considered.
Search strategies to identify studies for inclusion in systematic reviews often concentrate on identifying research using particular study designs and tend to be highly sensitive (retrieving a large proportion of the relevant research available) but low in precision (retrieving much additional but irrelevant research) (18). This process of thorough searching for information can be very resource intensive, requiring staff time for searching and sifting the results. There may also be costs for searching databases and acquiring potentially relevant records. Decision models for technology assessments are often undertaken within a relatively short period of time on a limited budget and have more numerous and varied data requirements than systematic reviews. The search techniques used in systematic reviews, therefore, may be impractical for the development of strategies to identify information to populate all the parameter estimates in a decision-analytic model. A compromise is required between ad hoc and unreported data collection that is often typical of decision model development and the thorough, highly sensitive search approach usually used to find effects evidence to inform systematic reviews.
One way to improve the quality of information gathering for decision models, without inundating researchers with large quantities of records, could be to conduct a range of focused searches with relatively high precision (a high proportion of relevant records retrieved with a relatively small number of irrelevant records). This strategy may offer a pragmatic solution that balances a clear systematic approach to research identification with an awareness of time and budget constraints. This study reports an investigation into the feasibility of carrying out focused searches to populate a case study model and explores the most efficient approaches to conducting such searches.
METHODS
Case Study
The case study chosen for this investigation was a decision-analytic model to assess the cost-effectiveness of prophylactic antibiotics for preventing urinary tract infections (UTIs) in children (5). The model was probabilistic, that is, parameter estimates were incorporated using appropriate distributions. Costs were assessed from a National Health Service (NHS) perspective, and benefits were expressed as quality-adjusted life years (QALYs). Simulation methods were used to determine the probability that alternative therapies were cost-effective at a range of threshold values which the NHS may attach to an additional QALY. Value of information (VOI) analysis was used to quantify the cost of uncertainty associated with the decision about which drug therapy to adopt. These features of the model determined the data requirements. In particular, prevalence and incidence data were required for the VOI analysis, and natural history data were required to model disease progression.
Developing Search Questions
To specify the information required to populate the parameters and their distributions in this model, a modeler and an information specialist developed 42 specific questions. These questions were complex and difficult to convert into search strategies. To simplify the searching process, the questions were grouped into five categories as follows.
a) Baseline Event Rates
These questions were subdivided into three categories: incidence and prevalence, rates of occurrence, and relationships between events in the model. The incidence and prevalence questions were related to the frequency of acute UTIs for children with recurrent UTI and no vesicoureteral reflux (VUR), mild VUR, or severe VUR and the incidence and prevalence of recurrent UTI with no VUR, mild VUR, and severe VUR. Second, rates of occurrence questions focused on identifying the proportion of acute UTIs that are pyelonephritic attacks in those with recurrent UTI and no VUR, mild VUR, or severe VUR. All these questions were asked for different subgroups of patients by gender and age. Finally, a set of questions focused on the relationships between progressive renal scarring, pyelonephritic attacks, end-stage renal disease, and significant consequences of end-stage renal disease.
b) Health-Related Quality of Life
The first question concerned the impact of acute UTIs and pyelonephritic attacks on the quality of life of infants and children. Two further questions focused on the reduction of quality-adjusted life expectancy (or life expectancy) as a consequence of end-stage renal disease and other consequences of progressive renal scarring.
c) Resource Use and Unit Costs
These questions addressed the costs of treating acute UTIs and pyelonephritic attacks in infants and children.
d) Relative Treatment Effects
These questions looked at the effectiveness of long-term low-dose antibiotics for UTIs and VUR, and the effectiveness of surgery in treating VUR.
e) Antibiotic Resistance
Information was required on resistance (in terms of reduced effectiveness) to trimethoprim, co-trimoxazole, and nitrofurantoin.
Selecting Appropriate Information Resources
There are many potentially useful resources for information to populate decision models (4;16;22). A small selection of these resources was assessed in this case study. Medical Literature Analysis and Retrieval System On-Line (MEDLINE) and Excerpta Medica Database (EMBASE) were searched for the majority of the parameter estimates, because these are large databases, widely available to researchers, and frequently cited in technology assessments. In addition to these databases, the Incidence and Prevalence Database (IPD) was used for questions relating to baseline event rates. IPD is a full-text database, which it was hypothesized might provide more efficient access to information than bibliographic databases, such as MEDLINE, whose records might not indicate that relevant information is contained in the full paper.
For questions relating to treatment effects, evidence was sought from research publications as close to the top of the hierarchy of evidence (18) as possible. Systematic reviews were sought from the Cochrane Database of Systematic Reviews (CDSR), the Database of Abstracts of Reviews of Effects (DARE), and the Health Technology Assessment (HTA) database. If no relevant systematic reviews were identified, the searches were extended to retrieve RCTs from CENTRAL (18). The NHS Economic Evaluation Database (NHS EED) and the Health Economic Evaluations Database (HEED) were searched for information about resource use and unit costs, because they contain economic evaluations from a range of sources (3;20).
Developing Search Strategies
The aim of the search strategies was to identify a few highly relevant records from a relatively small number of records. The full search strategies are published elsewhere (23) and were conducted between January and May 2003. The search strategies differed from the approach typically used for identifying evidence for systematic reviews of effects as, although both indexing terms and text words were used, the searches were limited to the most commonly used synonyms for the topic area, and only the most relevant subject headings were used. In addition, in some instances, indexing terms were limited to major subject headings (for details of major MeSH see http://www.ncbi.nlm.nih.gov/entrez/query/static/help/pmhelp.html), subheadings were used, and text words were limited to those occurring in the record title. This approach retrieved fewer records than the type of searches commonly used to identify research for systematic reviews. Inevitably, this focused approach can miss relevant studies as sensitivity is sacrificed for precision.
Search strategy development followed an iterative approach. If, in the view of the modeler, an excessive number of records was identified, the search strategy was adapted to become more focused and, thus, retrieve fewer irrelevant records, and if no relevant records were identified, the search strategies were adapted by replacing or adding terms.
The MEDLINE search strategies in OVID were translated to run in IPD using Dialog. Because of online costs, these searches on IPD were only run once. In addition, to reduce the costs of downloading records, an initial assessment of the records was carried out while online.
For the quality of life searches, three MEDLINE search filters were tested (2;21;22). These strategies, when used in MEDLINE, did not identify any records for the quality of life parameters. As the strategies were designed for MEDLINE and were also proving ineffective at retrieving utility information, converting these search filters for use in EMBASE was not considered appropriate. A fourth search strategy, which concentrated on identifying utility values rather than quality of life measures, was then devised (23).
RESULTS
The relevance of the search results was determined by the modeler in terms of the number of records that produced information to assist in the estimation of the model parameters. The search strategies retrieved an average of 51 records per question, but the number of records retrieved and the precision of the searches varied considerably across questions (Table 1). There was considerable overlap in the results of the searches both between databases and across questions (Table 1). Excluding the IPD results, 471 records were retrieved by more than one search strategy.
Although the three searches using published search filters (2;21;22) did not identify any records that were useful in terms of providing quality of life estimates, four of the records were relevant to other parameters in the model. Two were relevant to baseline events, one described resource use, and one was a clinical trial on the effectiveness of surgery.
DISCUSSION
Although based on a single case study, this research has raised several issues about the development of search questions, selection of appropriate resources, and development of search strategies to identify data for model parameter estimates.
Developing Search Questions
The information requirements to populate the model need to be well thought out and presented as answerable questions, which are applicable to the model. Carefully developed questions produced by discussion between the information officer and modeler should be more easily converted into searchable queries and more easily grouped by theme. Grouping by theme enables overlapping topics to be identified and searched together avoiding redundancy, reducing the number of searches, and saving time. In this study, despite grouping the 42 questions into 18 search strategies, there was still considerable overlap between the search results, indicating that further grouping of the questions would be feasible. A more-efficient approach to assessing the results of the searches might be to scan all the records together after deduplication rather than scanning the results question by question.
Selecting Appropriate Resources
This case study had too few relevant records to enable any general recommendations to be made about the most appropriate databases for each question. However, this study gives an indication of the sources or combination of sources that might be usefully explored.
a) Baseline Event Rates
EMBASE and IPD were useful sources for information on baseline event rates. Of the thirteen records identified as relevant to these questions, seven were indexed on EMBASE (five identified by the searches on EMBASE), and six on IPD (all identified by the searches on IPD).
The searches for baseline event rates in MEDLINE did not identify any relevant records. This finding may indicate the difficulty of identifying records on MEDLINE for these questions rather than indicating that MEDLINE does not contain any baseline events data, because five of the thirteen relevant records were actually indexed on MEDLINE.
IPD did not provide any information for the questions about the relationships between specific events in the model. It was, however, the most useful resource for the incidence and prevalence data and rates of occurrence, as these searches in IPD not only retrieved the greatest number of relevant records, but also gave the highest precision, 2.8 percent (6/212*100).
b) Health-Related Quality of Life
The results of these searches indicated the usefulness of searching both MEDLINE and EMBASE as unique records were identified on MEDLINE and EMBASE.
c) Resource Use and Unit Costs
HEED and NHS EED contributed an equal number of relevant records and unique records for the model, suggesting that both databases should be searched.
An additional relevant record was identified by the quality of life searches on MEDLINE (and this record is also indexed on EMBASE). This finding indicates that broadening the resource use and unit cost searches to MEDLINE or EMBASE would have produced further relevant records. However, the precision of searches on MEDLINE and EMBASE would probably be far lower than on HEED or NHS EED, as MEDLINE and EMBASE are much larger databases not restricted to economic records.
d) Relative Treatment Effects
CDSR, DARE, and CENTRAL were all useful in providing information to populate this parameter. An additional clinical trial was identified by the quality of life searches on MEDLINE (and is also indexed on EMBASE). If no relevant records had been found on these databases, MEDLINE and EMBASE are likely to have been useful in providing information from research with study designs lower in the hierarchy of evidence.
e) Antibiotic Resistance
An equal number of unique relevant records were identified from MEDLINE and EMBASE. However, for data on antibiotic resistance, using major subject indexing of the drug terms in MEDLINE seemed to offer more precise searches.
Developing Search Strategies
Although relatively few relevant records were selected (48 records), the search results did indicate how the search strategies might be improved:
a) Baseline Event Rates
These search strategies could have been focused further by limiting the results, where possible, to UK-based studies.
The relevant records indexed on EMBASE indicate how the search strategies could be made more sensitive. The search strategy for the consequences of end-stage renal disease could have included the terms “progression” or “follow-up” in the title field, and the search for the relationship between pyelonephritic attacks and progressive renal scarring could have included the floating subheading “complication” or the term “risk” in the abstract. However, adding terms to the search strategy to improve sensitivity would probably lower precision.
Although no relevant records were found by the baseline events searches on MEDLINE, five of the relevant records were indexed in MEDLINE. Two of these records, however, could not have been retrieved by the baseline events searches on MEDLINE: one was not indexed on MEDLINE at the time of searching, and one gave no indication that it contained any baseline events data. The other three records indicate that the term “progression” in the title field, the MeSH term RISK FACTORS, the text word “risk” in the abstract and the floating subheading “complications” might have increased the sensitivity of the baseline event rates search strategies in MEDLINE. However, more irrelevant records may also be retrieved by these search terms.
b) Health-Related Quality of Life
The published quality of life filters proved ineffective in this case study. There are three possible reasons for this. First, the published filters did not focus on the measures that are required for decision-analytic models, such as utility values. Second, very few studies have estimated utility values specifically for UTIs and related events. Finally, the optimal search terms necessary to capture records that describe these data have not yet been researched and tested.
Potentially useful search terms identified from the abstracts of the relevant records for the quality of life parameters areas follows: “utilities,” “index of well-being,” “QALM,” “quality-adjusted life month,” and “health status.” These terms could be tested in future searches alongside terms from other studies (22).
In this study, the searches for life expectancy data were particularly successful in terms of offering a relatively high precision. Searching the abstract for “life expectancy” was effective with three of the four records indexed in EMBASE and three of the four records indexed in MEDLINE having “life expectancy” in the abstract. Only two of these records in EMBASE and one in MEDLINE had LIFE EXPECTANCY as a subject heading and only one record in EMBASE and one in MEDLINE had “life expectancy” in the title. Sensitivity might have been improved by searching for the term “survival,” as this was in the abstract of three of the five relevant records.
c) Resource Use and Unit Costs
The results of these searches gave no indication how the search strategies could be further improved, although geographic limits could be applied to focus the searches.
d) Relative Treatment Effects
The relative treatment effects search strategies produced a relatively high number of relevant records with a reasonable precision. It was anticipated that these searches would be the least problematic, as the search approaches for effects data are well established. There are also relevant databases, such as those in the Cochrane Library, in which to conduct focused searches.
e) Antibiotic Resistance
There was no overlap between the relevant records found in MEDLINE and EMBASE. However, the two MEDLINE records were indexed in EMBASE and one of the EMBASE records was indexed in MEDLINE.
The two records missed by the EMBASE search strategy would have proved difficult to retrieve by focused searches, because one record did not contain EMTREE terms for the drug names and the other record only contained the EMTREE term DRUG RESISTANCE as a minor subject heading. These records were indexed very differently on MEDLINE. The record missed by the MEDLINE search strategy could not have feasibly been retrieved from MEDLINE, as it contained no MeSH terms for the drug names or for drug resistance. On EMBASE, this record was indexed with TRIMETHOPRIM, NITROFURANTOIN, and DRUG RESISTANCE. The differences in retrieval from MEDLINE and EMBASE, therefore, are due to different indexing practices. Focused searches on more than one database compensate to some extent for the loss of sensitivity of focused search strategies and the variations in indexing. In EMBASE, the major EMTREE subject headings, ANTIBIOTIC RESISTANCE, DRUG RESISTANCE, and DRUG SENSITIVITY seemed useful in terms of retrieving relevant records, and in MEDLINE, the MeSH heading DRUG RESISTANCE seemed useful.
CONCLUSIONS
Overall, the searches helped to identify important data for inclusion in the model. The modeler assessed 1,237 records for the 42 questions and 48 records contained data to populate the model, while further records provided useful background material.
Research Implications
This case study generated a small number of relevant records from each database. As such, it is difficult to generalize the findings to other decision models. More research is required to ascertain how far the approaches and strategies devised and tested in this case study are applicable and effective with other decision models. In addition, other resources and search approaches need to be tested. Further case studies should also explore if the records obtained produce different information in terms of both quantity and quality, and how these differences affect the model results and ultimately the decision-making process.
Policy Implications
Even using a pragmatic approach, conducting focused but systematic structured searches to populate decision-analytic models has time and budget implications that need to be considered when costing proposals. Costs include staff time, document acquisition, and, possibly, database charges.
The information specialist should ideally be involved with the project from the beginning. Decision-analytic models tend to be developed in a relatively short time frame, and many search questions and strategies need to be developed quickly. Therefore, discussions should begin at an early stage to allow adequate time to develop searchable questions and to construct and refine searches to define and populate the model.
The time and resources available to the project will ultimately determine the acceptable level of focus in the search strategies and which sources are searched. It is important to recognize that the identification of best available data for every parameter within the model may not represent a good use of resources (25). Although, highly sensitive searching for information for each parameter may not be feasible, the search process can still be systematic, transparent, and well documented. Searching can also be efficient, so that, given the resource constraints, more effort is spent on searching for those parameters, which are expected to have the largest influence on the results of the model (5;13;14;25). Decisions about the levels of focus in the search strategies and the resources used should be recorded, to enable readers to appreciate the compromises that have been made in identifying the evidence and, in principle, to enable the results to be reproducible. This information may also be useful to a decision-maker when assessing the accuracy of the model results.
CONTACT INFORMATION
Su Golder, BSc (Hons), MSc, Information Officer (spg3@york.ac.uk), Centre for Reviews and Dissemination, Julie Glanville, BA (Hons), MSc, PGDipLib, Information Services Manager, Centre for Reviews and Dissemination, Laura Ginnelly, BA (Hons), MSc, Research Fellow (lg116@york.ac.uk), Centre for Health Economics, University of York, YO10 5DD York, UK
This project was funded by the English Health Technology Assessment Programme, project number 02/32/01. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health of England.
