The West Midlands Health Technology Assessment Collaboration (WMHTAC) at the University of Birmingham is one of several academic units in the United Kingdom commissioned by the National Institute for Health and Clinical Excellence (NICE) to produce technology assessment reports (TARs) for the Department of Health. Historically, TARs have been concerned with the effectiveness and cost-effectiveness of drugs and devices rather than other types of interventions.
Recently, however, the Birmingham team undertook work to assess the effectiveness and cost-effectiveness of a psychological intervention—parent-training programs for the treatment of conduct disorders in children. This work was the first report in which the Social Care Institute for Excellence (SCIE) and NICE had worked together in this way. Involvement in a TAR on a social care–related topic meant that the WMHTAC team, including information specialist support, had to adapt their working methods accordingly.
AIMS
This case study arises from observations made while carrying out the literature searches for this TAR and subsequent retrospective examination of the search results. The focus will be on aspects of particular interest to information professionals, such as formulation of search strategies, most appropriate sources, and extensiveness of searches. The problems involved in trying to formulate searches to isolate specific study designs, in particular randomized controlled trials (RCTs), will also be addressed.
Experience of Searching in Social Care
Many of the smaller databases in social care were developed to meet a very specific organizational need and will differ in terms of software, search interface, and controlled vocabulary (if there is one). They may also lack the functionality of larger databases, such as the ability to export references in a structured format for reference manager software. Even in larger databases, searching may be difficult because of inconsistent indexing or inflexible search interfaces. Much of the social care literature is qualitative, and the research evidence is less likely to be in the form of RCTs. Furthermore, databases are unlikely to index papers according to research methodology. For these reasons, searching would be considerably less structured and straightforward than searching for information on the effectiveness of a drug or device, for which an established search protocol and methodology already exist.
METHODS
Formulating the Search Strategy
Study Design
At an early stage of the TAR, it was decided to focus on randomized controlled trials as the study design that would provide the best (least-biased) evidence when assessing the effectiveness of the intervention compared with placebo or another intervention. The range of studies to be included would be expanded only if RCT evidence were limited. A choice then had to be made about the most appropriate resources to search and how best to identify RCTs from them.
Sources of Advice
The review team included experts who had published systematic reviews on parent-training schemes and who provided valuable input on relevant sources and appropriate terminology. Another source of advice was the information team at SCIE, who were able to carry out searches of their CareData database on our behalf, using enhanced internal search facilities.
Selection of Resources
Input from team members and other experts provided a broader overview of resources available. Starting from the basic core of databases in the WMHTAC search protocol, the potential list of resources was expanded based upon advice received.
Hand Searching and Citation Searching
Given the overlap between subject areas (social care, health care, education, psychology) and the large range of potentially relevant journals from which to choose (the thirty-two trials examined came from a total of twenty-one different journals), it was decided that this approach was unlikely to be a fruitful strategy. (Note: Bibliographic details of the thirty-two RCTs selected for inclusion in the technology assessment report itself may be obtained from the authors on request.) Cochrane reviews on a similar topic did not perform any hand searching for the same reason and it was decided to follow suit, as the added value of searching one or two randomly chosen journals was likely to be very low. Citation searching was limited to the citations from good quality systematic reviews. No additional relevant trials were found in this way.
Implementing the Search Strategy
Adapting Strategies to Different Databases
The Cochrane Library, MEDLINE, and PubMed enable the user to increase specificity while searching by using the MeSH controlled vocabulary (U.S. National Library of Medicine's Medical Subject Headings), adding sensitivity by means of textwords where required. Such controlled vocabularies are less common in the social care literature and indexing can be inconsistent, making specificity in searching difficult. This finding proved a particular problem when trying to capture RCTs.
The sample search strategy in Table 1 illustrates how controlled vocabulary and (principally) textwords were combined with a methodological filter for the study design RCT in MEDLINE (2). In databases for which a controlled vocabulary was not available, a simplified combination of textwords was used to embrace the concepts of parent training and conduct disorder. RCTs were targeted using a basic combination of (trial* or random* or controlled), unless there was a means of limiting searches to type of research, as in the case of PsycINFO (by means of APA Online) with its limit “empirical study.”
Searching CareData
CareData was searched by SCIE by means of the in-house search interface using textword searches in title and abstract fields in an effort to retrieve relevant RCTs. Retrieval was then dependant on full research methodologies being recorded in either the title or abstract field. Due to the origins and purpose of CareData, early abstracts would not have consistently included full details of research methodology. Moreover, RCTs are not a research method commonly used in social care, so this again limited the number of records retrieved. Searches were then carried out to retrieve all research studies using a combination of controlled terms and textword searching, rather than limiting to RCTs. As attempts to specifically pinpoint RCTs cut down the number of references dramatically, the reviewers opted to scan the results of this broader search.
SCIE are currently in the process of re-developing the Electronic Library for Social Care (eLSC) and CareData. Changes planned include a structured taxonomy to enable the user to select the most appropriate term and the ability to search for specific types of content such as primary research.
Selection of Included Studies
The process of selecting the RCTs for inclusion in the TAR from the total search results is illustrated in the flowchart provided (Figure 1). Two reviewers examined abstracts and titles to identify potentially relevant studies with a third reviewer resolving any differences. Full texts of these identified studies were obtained and, again, the same process was applied to decide on the final thirty-two included RCTs. Contact with experts retrieved two additional trials but these trials are not considered in the case study as they were not located by means of database searches.
Flow chart of study identification. RCT, randomized controlled trial.
Testing the Search Strategies
Added Value of Additional Sources of Social Care Information
The first question to be addressed after completing searching for this TAR was how much more useful information was located by conducting such extensive searches? Recent methodological studies on literature searching for systematic reviews in health care suggest that searching beyond the “core” databases (Cochrane Library, MEDLINE, and so on) does not always locate sufficient extra information to justify the time expended (1;3). It was decided to examine retrospectively which databases yielded the thirty-two RCTs chosen for inclusion in the TAR. The individual Reference Manager databases for each bibliographic database searched were examined, and it was noted against each reference in which database(s) they were located. There was considerable overlap, but it was also noted that many of the databases appeared to provide no relevant references at all.
Success of Search Strategy
A combination of the core databases of Cochrane, MEDLINE, EMBASE, and PsycINFO together with input from subject experts and consultees enabled most published and unpublished RCTs to be located. However, it was decided to re-visit the searches later and address a second question—whether the social care and other supplementary databases searched did contain more of the thirty-two included studies—but these studies simply were not found by our search strategies and, if not, why not? Of particular interest was whether inclusion of study design terms within our strategies was not an effective approach on these databases.
Eight of the additional databases plus CareData were selected for examination, focusing on those that had a reasonably sophisticated search facility. Cross-checks were made using an author/title search to establish whether any of the thirty-two included references that had not been found by the strategy were nonetheless in those databases. The search strategy was then re-run to ascertain at which level a specific reference was not identified and whether in particular the terms used to pinpoint RCTs were at fault.
RESULTS
Formulating the Search Strategy
Selection of Resources
Table 2 illustrates how the list of sources to be searched expanded between protocol formulation and starting the main searches. Column 1 lists initial suggestions for the protocol based on experience of TARs plus knowledge of some social science/social care databases, column 2 lists those resources that were added after consultation with subject and social care information experts, and column 3 lists those resources suggested by these experts but ultimately not added to the list and the reason for that decision (in italics).
Thus, what was already a fairly lengthy list of resources became even more extensive. However, it was believed that, as this was the first technology assessment of this type undertaken by WMHTAC, some of the additional resources would be included and their usefulness examined retrospectively.
Implementing the Search Strategy
Added Value of Additional Sources of Social Care Information
An examination was made of Reference Manager databases containing the results of the searches to find which had yielded the final included RCTs (n = 32). Table 3 shows the main databases in which the key references were located. It would seem to illustrate what similar studies in health care information have suggested—a basic core of key databases will find most relevant references and searching any number of additional databases adds little or no research evidence (1;3).
Here, a combination of MEDLINE, EMBASE, the Cochrane Library, and PsycINFO would find all the thirty-two references. In terms of single databases, PsycINFO alone would have found 78 percent of the references with the Cochrane Library and EMBASE each locating 59 percent. Looking at databases in combination EMBASE and PsycINFO would have located 88 percent, the Cochrane Library and EMBASE 81 percent, MEDLINE and EMBASE 69 percent, and MEDLINE and the Cochrane Library 66 percent of the thirty-two selected RCTs.
Sixteen percent of the trials are unique references (four of these five references are from PsycINFO only, the remaining one from EMBASE). Databases from which the strategy failed to retrieve any of the thirty-two references were CINAHL, CareData, BEI, AEI, IBSS, SCI, NCJRS, and Sociological Abstracts. This finding is likely to be due to the content of these databases and the disparity between that and the type of material being sought but might also have been due to searching difficulties that will be examined later.
No references were selected from the results of the CareData searches. To cross-check why, the whole database was searched by author/title. It was confirmed that none of the thirty-two included studies were in CareData (i.e., neither were they found by the searches nor indeed were they in the database at all). This finding is probably explained by the coverage and content of CareData and also the qualitative nature of social care literature.
Performance of Search Strategy
Table 3 illustrates where the thirty-two RCTs included in the TAR were found. Duplicate references were found in a variety of different databases but, interestingly, many of the databases appeared to provide few or none of the references at all. The Methods section details how cross-checks were made to determine whether the references were in fact within the databases but had not been picked up by the search strategy and, if this was the case, at which level of the strategy they had been lost.
One striking feature of the results illustrated in Table 4 is that a more general search strategy could have located eighteen (56 percent) rather than eight (25 percent) of the thirty-two documents on SSCI. In the case of ASSIA, a more general strategy could have located six (19 percent) rather than one document (3 percent). Although fifteen (47 percent) rather than six documents (19 percent) were located on ZETOC using an author search, the original searches had been very general (three separate searches on terms for parent training only) due to the nature of the database and it would be unlikely to improve much on this particular search.
On examining the full records of the “missed” references, a picture emerged confirming that some references were missed because of indexing problems—for example, a paper that had been indexed under “training programs” rather than “parent training programs” would be lost to the searches. Some references were too old to have been indexed or include a searchable abstract—for example, those predating 1992 in the Social Sciences Citation Index do not contain searchable abstracts and, even after that date, only 60 percent of them do. One database did not contain any indexed items (ZETOC is a table of contents database) and if relevant terms such as parent training did not appear in an RCT's title then it would not be picked up when searching ZETOC (4).
Attempts to capture the study design RCT in non–health-care databases were not always successful either; if any of the text words contained in the basic RCT strategy did not appear in an article's abstract that item would not be retrieved. Records without an abstract were unlikely to contain any reference to study design in their title. Additional terms, such as “follow up,” “comparison,” “assigned,” could be used to expand the strategy, but this approach raises the question of trading off time against exhaustiveness, given the greater number of irrelevant references likely to be retrieved.
Strengths and Limitations of the Study
This case study has examined a topic that, to date, has received little attention, namely the problems encountered when carrying out literature searches for health technology assessments in social care and in particular the difficulties of focusing on a specific study design. It uses an actual case study of an HTA on parent training programs for treatment of conduct disorders in children to contribute to the ongoing methodological debate regarding how much added value extensive searching can bring.
Although every effort has been made to search strategically, it is likely that not all relevant RCTs were found and conclusions concerning where to search and how extensive searches should be are uncertain. In view of the breadth of the subject area, conclusions regarding search strategies for HTAs on social care topics in general are most uncertain. This case study is unlikely to be typical, given the large number of RCTs actually located and highlights the need for more such studies on a range of different social care topics as well as different study designs.
CONCLUSIONS
In a relatively new area for technology assessment, use was made of the knowledge of experts to guide the development of the search strategy. At the same time, it was considered important to use previous experience of health technology assessments to question whether further searching was likely to yield sufficient additional references of the required study type or quality to justify the time expended.
Recent papers on how extensively to search for information in the field of clinical medicine advise limiting searches largely to a core of main bibliographic databases (1;3). In the case of this particular social care topic, the yield resulting from the addition of more peripheral or general databases to the standard protocol only produced duplicate records (see Table 3).
Attempts to restrict the number of documents retrieved by using search strategies relating to specific study type did not seem to be particularly effective. This finding was confirmed by the discovery that some of the thirty-two RCTs were indeed in the social care databases but were not retrieved by the search strategies. However, the decision to search in this way was made because of time constraints and an awareness that the majority of relevant RCTs on the topic should have been retrieved by means of databases such as MEDLINE, Cochrane Library, and PsycINFO, which allow more specific searching. As it was, the reviewers still had an initial 3,857 documents to work with—this number would have been much greater had strategies to locate specific study type not been applied.
PsycINFO, a well-indexed subject-specific database, proved a valuable additional resource, retrieving several unique references. Although databases such as SSCI may have contained some of these references, the more sophisticated search facility in PsycINFO (Form/Content field) allowed them to be retrieved more easily and without large amounts of irrelevant additional data.
When moving beyond traditional health technology assessments, information specialists must become acquainted with both the unique features and the limitations of subject databases relevant to their topic. Where evidence-based research is not yet well established, it is not always easy, or indeed useful, to try and replicate searches based on study design. In many respects, however, the procedures to be followed will be the same whatever the topic—break the subject down into an answerable question and translate it into a search strategy, where discussion with subject specialists can be a useful guide to terminology and concepts. Select the most appropriate resources to be searched, given time and budgetary constraints, and ensure that searches are carefully documented so as to be reproducible.
An area for future research would be to investigate further the problems of searching for good quality studies and specific study designs in a range of social care topics. In time, as evidence-based research becomes more embedded in social care, resources and their search capabilities are likely to be improved, as the ongoing work on CareData illustrates, and this improvement too will facilitate the information specialist's task.
CONTACT INFORMATION
Susan E. Bayliss, BA (s.bayliss@bham.ac.uk), Information Specialist, Janine Dretzke, MSc (j.dretzke@bham.ac.uk), Systematic Reviewer, Department of Public Health and Epidemiology, WMHTAC (West Midlands Health Technology Assessment Collaboration), University of Birmingham, Edgbaston, Birmingham B15 2TT, UK
Grateful thanks to Karen Winchester of Social Care Institute for Excellence for her assistance with the CareData searches and for her insight into searching on social care topics and the proposals to develop CareData. Thanks also to Clare Davenport and Anne Fry-Smith of the Department of Public Health and Epidemiology, University of Birmingham, for reading this paper and offering constructive comments and suggestions and to Jane Barlow (Primary Care Career Scientist, University of Oxford) and Sarah Stewart-Brown (Professor of Public Health, University of Warwick) for their suggestions during the formulation of the search strategy.
Note: Bibliographic details of the thirty-two RCTs selected for inclusion in the technology assessment report itself may be obtained from the authors on request.
This work was carried out under West Midlands Health Technology Assessment Collaboration's contract with the United Kingdom (UK) National Health Service, and as such, this article is considered a work of the UK government and not subject to copyright within the United Kingdom.
