Background
The first step of an HTA-informed decision process involves identification and prioritization of the health technologies to be assessed. The most recent definition of HTA specifies its role in assessing the value of health technologies throughout their life cycle (Reference O'Rourke, Oortwijn and Schuller1). In the context of HTA, horizon scanning is the systematic identification of health technologies that are new, emerging, or becoming obsolete, and that have the potential to affect health, health services, and/or society (2). Similarly, in HTA, disinvestment is defined as “the deliberate and systematic reduction of funding for a health technology of questionable or comparatively low value (3).” In an era of growing economic pressures on all health systems, HTAs are needed to inform decisions on both investment and disinvestment in health technologies. An efficient practice for prioritizing topics for HTA should contribute to reduce delays and inequalities in the availability of innovative health technologies and reduce nonefficient practice. Priority setting in HTA has been a topical issue for decades and several reviews of practices have been published (Reference Henshall, Oortwijn, Stevens, Granados and Banta4–Reference Specchia, Favale, Di Nardo, Rotundo, Favaretti and Ricciardi7). Prioritization of topics for HTA is highly context-dependent, and different methods have been developed and explored in different settings (Reference Lepage-Nefkens, Douw, Mantjes, de Graaf, Leroy and Cleemput8–Reference Polisena, Trunk, Gutierrez-Ibarluzea and Joppi12). In the field of horizon scanning to inform topic identification in HTA, a toolkit (2) produced by EuroScan International Network (now International HealthTechScan, i-HTS [www.euroscan.org/]) has been most commonly used. Also, tools such as the PriTec online tool developed by the Galician HTA agency based on predefined criteria are commonly used to prioritize topics for HTA (Reference Ruano Raviña, Velasco Gonzalez, Varela Lema, Cerdá Mota, Ibargoyen Roteta and Gutiérrez Ibarluzea5;Reference Varela-Lema, Rozas, del Carmen, Preito Yerro, Arriola Bolado and Asua-Batarrita13). In the area of disinvestment, an evidence-based framework for identifying topics of low value has been recently published (Reference Esandi, Gutiérrez-Ibarluzea, Ibargoyen-Roteta and Godman14). However, there is currently no consensus on what constitutes best practices for identification and prioritization of topics to be assessed in a particular context. Sharing scientific publications on methods related to horizon scanning and disinvestment is an important best practice. Ideally, it should reduce duplication of work for those interested in developments and innovations in horizon scanning and disinvestment.
At the 2017 HTAi annual meeting, in Rome, a collaborative current awareness alert service was proposed to the Disinvestment and Early Awareness Interest Group (DEA-IG). The possibility of expanding an earlier informal alert service for the interest group was discussed further at the 2018 HTAi annual meeting, in Vancouver. A proposal was submitted to the HTAi for a 1-year pilot project, which was subsequently accepted in January 2019.
The intent of the DEA-IG alerts was to identify new studies on disinvestment and horizon scanning, particularly new research and methods studies, and broad forecasts of technologies that may be of interest to those involved in HTA, horizon scanning, and disinvestment initiatives. This collaboration of members of the DEA-IG and IR-IG aimed to increase efficiency in HTA agencies and others working in the field of horizon scanning and disinvestment to prioritize topics for HTA. This was done by exploring search strategies for iterative dissemination of citations for new papers with information on tools and methods relevant to the DEA-IG members.
Methods
Members of the DEA-IG and IR-IG from the Norwegian Institute of Public Health (NIPH), the Canadian Agency for Drugs and Technologies in Health (CADTH), and the DEA-IG steering group held initial conference calls in the fall 2018 to discuss the process for developing an alert. An action plan was developed in January 2019, and in May 2019, the first issue of the alert was developed and used as a prototype. The template for the alert was based on an e-newsletter previously developed by the IRIG (15). Search strategies were developed by CADTH and NIPH information specialists. Each alert was three to four pages in length, included the citations and links to the full papers or to a PubMed abstract if the full text was not freely available, and was developed every quarter. Staff at the NIPH and CADTH selected and reviewed the potentially relevant items for each issue. The final version was sent to the HTAi Secretariat, which posted it on the DEA-IG Web site and circulated an email announcement to the members of the DEA-IG (approximately 330 individuals), with a link to the Disinvestment and Early Awareness Alerts.
Search Strategy and Inclusion of Citations
The PubMed searches were drafted in March 2019 and set to run automatically. No language or other limits were applied. The Google Alerts searches were also set up automatically to run weekly. An English language limit was applied to both Google searches, and Web sites ending with “.com” were excluded using the “-” operator for the horizon scanning alert. This was intended to exclude commercial content. Google Alerts allows only a relatively small number of keywords, and consequently, the search terms were very basic. The final PubMed and Google search strategies are available in the Supplementary Material.
Additional sources of information, such as new studies identified by the authors, colleagues, or through routine horizon scanning work, such as the annual Horizon Scan Roundup prepared by the CADTH (16), were also included. Search results were independently reviewed by at least two authors at either the NIPH (the horizon scanning search) or CADTH (the disinvestment search) based on the citation's title and abstract. Inclusion criteria were set to be papers on methodological issues, topics, or broader forecasts relevant for disinvestment and prioritization of topics for HTA. No restrictions were made to the type of publication. Disagreements were solved by consensus. The included citations were then categorized by at least one author at CADTH according to being most relevant for horizons scanning or disinvestment. The final list of included and sorted citations for each alert was checked by at least one author at the NIPH.
Recall and Precision Rates
Knowing recall and precision rates of search strategies is useful for future revisions and validations and can be explored based on defining gold standards (Reference Massonnaud, Kerdelhué, Grosjean, Lelong, Griffon and Darmoni17). However, a true gold standard of relevant citations is not available in this case. As we observed that additional relevant PubMed citations were identified through other sources, we decided to use a pragmatic definition of gold standard where all relevant PubMed indexed citations were defined as the total number of relevant citations that should ideally have been identified by the searches during the project time frame. A rerun of searches for the total period from 1 January 2019 to 31 August 2020 was performed. Based on the PubMed index number available for alerts issues 1–6, we checked if the citations were retrieved by the searches.
Retrospective Sorting of PubMed Citations
In addition, to investigate what type of information was identified by the PubMed searches, all PubMed citations were retrospectively sorted based on the following: study type (primary study, systematic review or a systematic scoping review, or other); being a broad forecast on technologies; a resource for information on relevant methods, tools, or networks; technology type (pharmaceutical, medical device or in vitro diagnostics, other, or mixed); medical specialty (oncology, cardiology, endocrinology [including diabetes], other, or mixed). Each record was also evaluated according to its relevance for HTA. To be defined as highly relevant to HTA, HTA had to be mentioned in the abstract or full text. If not considered high value, a judgment between low and unclear was made based on information available in the abstract or full text. The judgment and sorting was done independently for all references by two authors (VL and ALE). Full-text versions were inspected only if the abstract did not provide enough information. In cases of disagreement, the reference was checked by one author (VL) who made the final decision.
Survey
An evaluation at the end of the pilot phase was conducted to determine the usefulness of the alerts and whether the project should continue. The evaluation included information on downloads of the alerts from the Web page, informal feedback on the alerts during the pilot, and a survey of the DEA-IG members.
The survey was developed in SurveyMonkey and sent via email to all members of the DEA-IG (330 individuals). The survey was distributed in September 2020. Recipients received a reminder after 2 weeks.
Results
Identified Records
A total of six issues of the alerts were produced and disseminated on the DEA-IG Web page (18) during the project time frame from June 2019 to September 2020. Each alert contained between 20 and 39 new citations sorted by publication type (journal article or papers/reports/books). PubMed indexed citations were further sorted according to their relevance for disinvestment or horizon scanning. The PubMed searches retrieved relevant papers for every alert, and these formed the bulk (124 of 170 citations) of information in each issue. Of the 124 unique PubMed citations identified in the alerts, 70 references related to horizon scanning, and 54 related to disinvestment (see Table 1). Other papers and reports (i.e., nonjournal articles) were not categorized. There was some duplication of references in the search results; for example, when a PubMed record was first captured for an alert, a publisher could have supplied reference or an electronic publication ahead of the print journal issue, and then this was captured again by a subsequent alert when it was indexed for MEDLINE.
Table 1. Numbers of new publications in each alert
a Most of these were identified through other work at the Canadian Agency for Drugs and Technologies in Health, CADTH.
D, Disinvestment; HS, Horizon Scanning.
Retrospective Sorting of Retrieved PubMed Citations
Of the seventy citations included as relevant for horizon scanning or early awareness, we found that seventeen could be classified as primary studies and sixteen as systematic reviews or systematic scoping reviews. The rest were either editorials, commentaries, or reviews without a reported systematic search strategy. Thirty-one papers relevant for horizon scanning could be categorized as broader forecasts of technologies, and fourteen contained information on tools, methods, or networks. Twenty-six citations relevant for horizon scanning specifically mentioned HTA.
Of the fifty-four citations retrieved as relevant for disinvestment, we considered twenty-one to be primary studies and eight to be systematic reviews or systematic scoping reviews. As for horizon scanning, the rest were either editorials, commentaries, or reviews without a reported systematic search strategy. Only six citations relevant for disinvestment could be categorized as broader overviews of technologies. We considered that fifty-one citations contained some information on tools or methods relevant for disinvestment. Only nine specifically mentioned HTA. Table 2 provides the main results of retrospectively sorting the 124 PubMed citations based on abstract and, where available, full-text review. All citations and the full result of sorting are available in the Supplementary Material.
Table 2. Study type and subject areas of PubMed-indexed references
a Mainly editorials, commentaries, or reviews without a reported systematic search strategy.
HTA, health technology assessment
Recall and Precision Rates
Knowing recall and precision rates of search strategies is useful for future revisions and validations. We found that the Google results mostly retrieved industry market-research forecasts (despite the search restriction on “.com” sites), nonrelevant general health or business news, and information from fields outside of health care, rather than studies and reports from other HTA agencies, government, or research groups.
To determine how the search results fared with the pragmatically set gold standard, the PubMed searches were rerun after the pilot project period. The rerunning of the final search strategies for the pilot project period 1 January 2019 to 31 August 2020 retrieved 1,976 records for disinvestment and 1,895 records for horizon scanning. For the horizon scanning search strategy, the recall rate was .7 (49 out of 70) and the precision was .025 (49/1,895). For the disinvestment search strategy, the recall rate compared with the total number of identified relevant PubMed records was .92 (50 out of 54) and the precision was .025 (50/1,976).
Resource Requirements
In terms of resources required, once the initial search strategies were developed, the staff time required to produce the monthly alerts was approximately one work day per alert at each of the collaborating agencies (the NIPH and CADTH)—mainly for scanning through the search results for relevant publications and for compiling, formatting, and editing each new issue.
Web Page Downloads
Web page downloads, provided by the HTAi Secretariat for the first five alerts (as of September 2020), indicated a small, but consistent number of downloads of each issue, ranging from 32 to 56 downloads per issue. The issues were downloaded from users in various countries in Europe, North and South America, and Southeast Asia.
Informal Feedback
Discussion of the first issue of the alert at the DEA-IG business meeting, in June 2019, did not garner any comments regarding the format, value of the information, level of detail, or frequency of the publication. An informal request for reader feedback included in the third and fourth issues of the alert resulted in four responses—all positive, and some with suggestions for expansion of the alerts. The readers’ comments are provided in the Supplementary Material.
Survey
Seventeen survey responses were received from the DEA-IG members (5.6 percent response rate). The low response rate is similar to that reported in a recent survey of disinvestment activities at HTA agencies (a 7.5 percent response rate) (Reference Polisena, Trunk, Gutierrez-Ibarluzea and Joppi12). In addition to consent to use their information, the survey included eight questions. The key results are shown in Figure 1. Approximately 90 percent of respondents found the alerts useful and the majority were interested in both disinvestment and horizon scanning. Keeping up to date was the main use of the alerts (82 percent). More than 80 percent of the respondents reported working in HTA; around 50 percent worked with disinvestment or reassessments; about the same number responded to working with horizon scanning. Regarding whether they had looked at the alert and how often they did so, fourteen respondents (82.3 percent) noted that they had taken the time to look at the alerts, three respondents (17.6 percent) had not yet looked at them, eight respondents (47.0 percent) checked the alerts whenever a new one was issued (i.e., quarterly), and six respondents (35.2 percent) looked at them occasionally. The final question was an open-ended request for feedback or suggestions for improving the alerts. Six comments were provided—four expressed satisfaction with the alerts, one suggested changing the column layout, and one suggested more focus on identifying technologies for disinvestment.
Figure 1. Usefulness of the alerts and main uses. Response rates on usefulness of the alerts and use of the alerts from respondents who had looked at the alerts. Other uses included to inform current activities and for up-to-date literature and research activities.
The alerts have also been useful for the collaborating agencies themselves. For example, the NIPH has used the alerts to identify information in planning of a disinvestment project in Norway, for the European Network for Health Technology Assessment (EUnetHTA) Topic Identification, Selection and Prioritization recommendations, in planning a project on horizon scanning in LMICs, and for facilitating a workshop for WHO in the Balkan region. At the CADTH, the alerts are circulated to other staff involved in horizon scanning, and selected references from the alerts are used in the annual CADTH Horizon Scan Roundup (16).
Discussion
This pilot collaborative project between DEA-IG and IR-IG produced six alerts with a total of 170 citations for publications relevant to disinvestment and early awareness or horizon scanning. These alerts were disseminated to members of the DEA-IG from June 2019 to September 2020. The main aim of the pilot was to share papers on tools or methods relevant for the field, rather than sharing lists of relevant topics. As broad forecasts or overviews of relevant technologies are methodological relevant, these were included.
A total of 170 citations of which 124 were PubMed-indexed, covered the disseminated citations during the pilot. The PubMed search strategies retrieved 99 out of the 124 PubMed-indexed citations, with .92 recall rates for the disinvestment strategy and .7 for the horizon scanning strategy. Precision was .025 for both. Additional PubMed citations and most of the gray literature reports were identified through other sources, in particular the Horizon Scan Roundup performed by the CADTH (16). Google alerts provided very few relevant publications. Google alert searches are quite limited in the number of search terms or characters that can be used and in the search capabilities in terms of combining terms and truncation (Reference Hill19). Search algorithms for Google favor business and market intelligence over government, and research agency reports were not found to be useful for either horizon scanning or disinvestment searches.
As various terms are used for the concepts of disinvestment (Reference Gnjidic and Elshaug10;Reference Niven, Mrklas, Holodinsky, Straus, Hemmelgarn and Jeffs11) and horizon scanning (Reference Varela-Lema, Punal-Rioboo, Accion, Ruano-Ravina and Garcia20), the search terms of the PubMed search strategies were broad—developed to maximize recall relative to precision. Consequently, most of the papers identified in the PubMed searches were not relevant.
Notably, the recall and precision numbers are far from 100 percent correct, as we do not know the true number of relevant articles in PubMed at the given time point. We only know the total number in the alert issues 1–6. Nevertheless, the results reflect that The PubMed search on horizon scanning did not produce satisfactory results. For instance, in issue 2, we noted that there were more relevant references picked up by the regular CADTH horizon scanning work compared with the PubMed search. As there is a delay and lack of consistency in indexing Medical Subject Headings (MeSH) terms, one explanation could be that the search strategy used for issue 1 relied quite heavily on MeSH indexing. To mitigate this, we added additional text words searching the title field only. However, as reflected by the final recall rate of .7, this did not solve the problem completely. The search strategy for disinvestment had a higher recall rate. This could reflect that other sources were mainly focused on horizon scanning. We consider that a continuation of the work should include attempts to revise and validate the search strategies to increase both the recall and the precision rates. For this work, the identified references could be used as a validation set, and the results should be relevant for anyone who wishes to identify and monitor new research relevant for disinvestment and horizon scanning. As identification of duplicate references was conducted manually, the use of a bibliographic software package to manage references and remove duplicates may have been worthwhile.
A 6 percent survey response rate indicates that a relatively small number of members of the DEA-IG are currently using the alerts. Nevertheless, the majority of those who did respond expressed that the alerts were useful for their work. We have noticed that some public horizon scanning programs in HTA have been scaled down in recent years. This includes the horizon scanning programs at the Agency for Healthcare Research and Quality (AHRQ) that were provided by the nonprofit foundation, ECRI (Reference Duda, Fleming, Kirwan, Roff and Rich21), in the U.S., and the Australia and New Zealand Horizon Scanning Network (22). If this reflects a decreased role of horizon scanning or local priorities and resource issues could be further explored by searching for or conducting surveys on HTA prioritization processes in different contexts. Notably, ECRI is still providing horizon scanning services in the U.S., and the CADTH horizon scanning program, in Canada, has continued since its inception in 1998 (https://www.cadth.ca/about-horizon-scanning).
The retrospective sorting revealed that high relevance to HTA (that is HTA stated in the abstract or paper) was limited to thirty-five publications. These thirty-five references could be reviewed more carefully to identify recent methods and tools relevant for the field and specific contexts.
Possibly, lists of obsolete technologies and new technologies may be more relevant for the DEA-IG members than searching for scientific publications on methods and tools. The retrospective sorting of PubMed citations revealed that the number of broad forecasts or overviews of technologies was low for disinvestment. This may reflect that updates on new and emerging technologies as well as obsolete technologies are primarily not found in scientific journals. Disinvestment has gained more attention in the HTA community recently, with the focus on methods and research papers in the alerts. However, interest on the identification of specific technologies for potential disinvestment may not be as relevant for publishers. Priority setting in HTA is the process leading to a decision on which topic to assess given the limited assessment resources available (Reference Henshall, Oortwijn, Stevens, Granados and Banta4). Priority setting in HTA is extremely important, but it is highly context-dependent. Identification and dissemination of publications on new methods to efficiently identify and prioritize topics for HTA in different contexts may benefit from automatization and better dissemination of the information and through increased cooperation between HTAi interest groups, international and regional collaborative initiatives, and scientific networks such as the i-HTS. If timely dissemination of lists of technologies is of higher importance, additional methodological approaches and dissemination strategies need to be explored. For example, if relevance for Low- and Middle-Income Countries is aimed at, future alerts for HTAi members should be developed in collaboration with the HTAi Interest Group for Developing Countries.
Conclusion
Current awareness alerts were produced through a collaboration of two HTAi interest groups, the DEA-IG and the IR-IG, to provide researchers and HTA agencies with recent publications on the topics of disinvestment and horizon scanning. Based on email feedback, downloads, and a formal survey, the alerts were considered useful, but due to the low number of survey respondents, these findings were limited in nature. The results of this pilot project can be used to revise search strategies and information sources, improve the relevance of the alerts, and plan for expanded dissemination strategies.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462321000568.
Acknowledgments
We would like to acknowledge Dr. Julie Polisena, former Chair of the HTAi Disinvestment and Early Awareness Interest Group, and Amanda Hodgson, Manager, Research Information Services at the CADTH, for their assistance in initiating this project.
The authors gratefully acknowledge the support of the HTAi Secretariat, NIPH, and CADTH for this pilot project.
Funding
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflicts of Interest
There are no conflicts of interest.
