Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-02-06T10:21:41.544Z Has data issue: false hasContentIssue false
  • English
  • Français

Time-trends in health technology assessments: An analysis of developments in composition of international health technology assessments from 1989 to 2002

Published online by Cambridge University Press:  25 October 2005

Eva Draborg  and
Dorte Gyrd-Hansen
Eva Draborg
Affiliation:
University of Southern Denmark
Dorte Gyrd-Hansen
Affiliation:
University of Southern Denmark
Rights & Permissions [Opens in a new window]

Abstract

Objectives: Health Technology Assessment (HTA) as a method for producing evidence in the health-care sector has been used for more than 25 years but has grown in extent during the past years. The objective of this study is to explore a possible evolution in these HTAs, in type of assessed technologies, in type of assessors, and in its methods.

Methods: A structured literature review was conducted of 433 HTA reports from the period 1989 to 2002 by eleven leading HTA institutions worldwide. The review focused on the methodology used, the assessors, and the assessed technologies and was designed to elucidate general time-trends in the practical application of HTA.

Results: The study shows that literature reviews are still the most often used method of assessment and accounts for a relatively stable fraction of assessments. The fraction of economic evaluations shows a slightly decreasing trend in contrast to randomized controlled trials and modeling/evaluations, which are applied more frequently. The data also demonstrate a more frequent use of external partners as assessors and a shift between devices and pharmaceuticals as assessed technologies.

Conclusions: The study shows an increase in the number of HTAs but no major developments in assessment methods used and, therefore, no widespread spillover from the development in research methods in general to the field of HTA methodology.

Keywords

HTA methodsType of technologyAssessorsInternational time-trend
Type
GENERAL ESSAYS
Information
International Journal of Technology Assessment in Health Care , Volume 21 , Issue 4 , October 2005 , pp. 492 - 498
Copyright
© 2005 Cambridge University Press

Health Technology Assessment (HTA) was originally designed more than 25 years ago with the purpose of assessing the consequences of medical technologies and was primarily based on the synthesis of existing evidence (22). HTA has its beginnings in the United States, with the establishment of the Office of Technology Assessment in 1972, and has since then spread to Canada, Australia, and Europe, with Sweden and the Netherlands as some of the first HTA agents, and later to many other countries around the world (1;3;19). As of today, a total of forty-two HTA institutions from twenty-one countries are members of the International Network of Agencies for Health Technology Assessment (INAHTA), and the number of members of Health Technology Assessment International (HTAi) after 1 year in service is sixteen organizations. These numbers alone indicate a high level of activity in HTA internationally in 2004.

From the beginning, HTA has been defined by its purpose not by its methodology (16) and the primary method of assessment was review of existing evidence. The reviews were later developed into systematic reviews, and consensus methods were used (13). Today, systematic reviews are still the most-often mentioned method of assessment when looking for descriptions of HTA methods in the literature (4;8;14). Primary research is also included as constituting HTA methodology but most often as a supplement to systematic reviews when evidence is lacking in some areas (21;23).

Despite this uniformity in descriptions of the underlying methods of HTA, time-trend comparisons may well be warranted to disclose any general trends in the practical applications of HTA. Some authors have pointed to the need for methodological developments in HTA (19), others have noticed variations in HTA methodology between HTA agents (15). Several authors see such variations as a potential problem (23;26), and the need for international collaboration in dealing with and defining HTA has been stressed (1;18).

International collaboration has taken place in terms of the EUR-ASSESS project in the late 1990s (16), the ECHTA/ECAHT project in early 2000s (4;15), and the INAHTA initiatives in the early 2000s (9). All of these projects have focused on HTA methodology, and attempts have been made to reach consensus about the content of HTAs as well as the methods applied in HTA. Attempts at the latter have had limited success, most likely because HTAs are applied in different countries with different (political) cultures and because HTAs rely on methods from different scientific fields. Some of the participants involved in the aforementioned initiatives conclude that HTA methodology cannot be standardized and that one methodological approach to HTA does not exist (16;26). It is noteworthy that the INAHTA Checklist developed with the purpose of improving the standards of HTA focuses mainly on the content of HTA, whereas methodological issues are almost absent.

Considering the speed of development within the scientific fields that underlie HTA, the methodologies applied in HTA are likely to have changed over the past decades. Within the past 25 years, there has been a considerable growth in HTA activities due both to the expansion in number of HTA agents and to the production levels of these agents (6). The scope of HTA has also widened from primarily addressing effectiveness and safety issues to covering a broader array of issues such as psychological, organizational, ethical, and legal aspects (6;14;23). All of this gives rise to questions like Have the methodological developments in general spread to HTA methodology? Who are the assessors of the increasing number of produced HTAs? And what type of technologies are the subjects of the increasing numbers of assessments?

The objective of this study is to explore a possible evolution in HTA methods, in type of assessed technology and in type of assessors during the period 1989–2002 to disclose the extent to which the original focus of HTA may have changed and to disclose possible further developments in HTAs worldwide. To these authors' knowledge, no such study of the international time-trends of HTA has been published previously in the literature. Only Menon and Topfer (20) focused on time-trends, but the scope of their study was restricted to analysis of 117 government-funded Canadian HTAs. A few other studies exist that focus on HTA methodology, but none of them have included a time perspective. Perry et al. (24) performed a study of HTA reports originating from 103 HTA institutions in 24 countries. The weakness of this study was that it was not based on actual review of the HTA report but relied on the HTA institutions answers to a questionnaire. Furthermore, the authors applied a very broad definition of HTA institutions to also include medical societies, for-profit organizations, and trade associations. Mears et al. (18) also relied on questionnaire data but restricted the respondents to fifty nonprofit and/or government-financed institutions. Poulsen (25) carried out an actual review of HTA reports produced from a sample of HTA institutions applying inclusion criteria similar to these presented in this study, but the time period was limited to 1989–1996, and no time-trend analysis was performed. Finally, García-Altés et al. (7) reviewed sixty-one HTA reports but restricted their sample to four HTA organizations, restricted their study period to 1999–2001, and had no time perspective in their analysis.

METHODS

A structured literature review of Health Technology Assessment reports published from leading HTA institutions in the period of 1989–2002 was performed. The sample consists of 433 HTA reports from 11 HTA institutions in 9 countries, which are ASERNIP-S and AHTAC/MSAC from Australia, CCOHTA from Canada, DACEHTA from Denmark, Health Council from The Netherlands, NZHTA from New Zealand, SMM from Norway, SBU from Sweden, NCCHTA from United Kingdom, and AHQR and VATAP from the United States. To be included in the study, the HTA institution had to be national, nonprofit, and noncommercial. Furthermore, to secure a certain level of experience, we only included institutions with at least 4 to 5 years of experience in the field of producing HTAs and at least nine published HTA reports. The institutions were traced from lists of members of INAHTA (10) and ISTAHC (11) and were cross-checked with a list of HTA resources on the Internet (5) and with publications from the Health Technology Assessment Database (12).

Selection criteria for inclusion of the individual HTA report were that it had to constitute a full HTA; first, according to the institution's own characterization and, second, by judgment of the reports separately. This procedure excluded all technology reviews, short reports, early warnings, journal articles, and methodological reports. Finally, the HTA reports had to be written in English or one of the Scandinavian languages, and it had to be obtainable by the institutions home page or by written contact to the institution.

The HTA reports were obtained by searching the home pages of the relevant institutions, and full lists of all HTA reports were inspected. When a certain report was judged relevant, it was printed from the home page. Alternatively, the institution was contacted by e-mail or at their postal address, and a printed copy of the report was requested. This process resulted in an almost complete sample of HTA reports from the selected institutions (n = 433). In few cases, two individually published reports were interpreted as representing one HTA (for example, when the heading was “Part One. Technological Review” and “Part Two. Economic Evaluation”).

In the reviewing process, we used a predetermined checklist with special focus on methodology supplemented by general information on the reports. The checklist consisted of questions describing the individual report, its institution of origin, year of publication, type of assessed health technology, type of assessor(s), and methods of assessments.

Type of health technology was coded according to the definition used in Banta and Luce (2) where a pharmaceutical is “any chemical or biological substance that may be applied to, ingested by, or injected into humans,” a device is “any physical item, excluding drugs, used in health care,” and a procedure is “a combination of provider skills or abilities with drugs, devices, or both.”

Assessment methods were coded according to the individual description in the reports, and they were grouped in seven general methods of assessment (literature review, meta-analysis, survey, randomized controlled trial (RCT), economic evaluation, modeling/evaluation, other method). Multiple assessments were found in a large proportion of reports, and the individual assessments were coded individually, resulting in a total number of 682 individual assessments in the 433 HTA reports.

The assessment method literature review contains all assessments for which some kind of literature review was stated in a report, both systematic and less formalized reviews such as narrative reviews of the subject in question. Meta-analysis was coded when results of a formal meta-analysis were presented and were designated as such in a report together with at least a description of data and methods of analysis. Surveys were coded as present in a report in the cases where it was described together with data and survey methods used. Both interviews and questionnaires were included in this method of assessment as long as it included new data and quantitative analysis. Randomized controlled trial was judged to be present in a report when it was stated and the data, procedure of randomizing, and method of analysis were presented in the reports. The assessment method economic evaluation contains all assessments that were labeled as economic evaluations by the authors and includes types of evaluations such as cost-minimization, cost-effectiveness, cost-utility, or cost-benefit analysis. The term modeling/evaluation included assessment methods based on different kind of modeling techniques, for example, decision analysis, simulation models, and so on, and of less-structured evaluation studies not to be included in the categories of surveys and RCTs. The subjects of modeling or evaluation studies were not confined to the clinical parameters but could concern other parameters too, for example, economic parameters. The last method of assessment, other method was coded as being present when an HTA contained analysis not covered by one of the former six assessment methods. Among these coded were formal methods of synthesis such as consensus conferences and Delphi techniques and assessment methods such as case reports, expert opinion, and so on. To simplify the presentation of results, the data are grouped in four time periods (1989–1993, 1994–1996, 1997–1999, 2000–2002).

RESULTS

The Production of HTA Reports in 1989–2002

Table 1 displays the number of HTAs by institution in the period 1989 to 2002. NCCHTA in the United Kingdom is by far the most productive HTA institution in the sample, publishing almost 30 percent of the total number of HTAs.

Figure 1 shows the number of HTAs by year of publication and illustrates the significant increase in the number of HTAs during the time period 1989–2002. A total of 52 percent of the reports in the sample are published in the period 2000–2002, which means that almost the same share of HTAs in the sample is published in the past 3 years as in the first 11 years of the study period. This development reflects an increase in the number of institutions producing HTA reports as well as an increase in the number of reports produced by institution.

Number of health technology assessment reports by year of publication

Type of Assessed Technology

During the study period, health procedure is consistently the most assessed type of technology (Table 2), constituting more than 50 percent of all assessed technologies in all four time periods. Assessments of medical devices were salient in the early 1990s and constituted more than 30 percent of the assessed technologies in the period before 1996. After this period, the share decreased. Conversely, the fraction of HTAs with a focus on pharmaceuticals was very low in the early time period, but since the year 1996, the fraction has increased to more than 1/4 of all assessed technologies.

Type of Assessors

Over the study period as a whole, the HTAs are equally frequently performed by the HTA institution's own personnel, external partners, and a combination of the two (Table 3). However, over time, a major shift has taken place. During the first 6 years of the study period, almost all of the HTAs were assessed by the HTA institutions' own personnel (around 90 percent). External partners and joint assessments were only sporadically observed, but from 1996, the use of external partners became more common, and from around 1999, joint assessments became frequent, leaving less than 10 percent of the HTAs assessed by their own personnel alone.

Methods of Assessment

Table 4 shows the frequencies of seven different types of methods of assessment applied in the 433 HTAs. The number sums to a total of 682 observations because more than one assessment method could be used in one HTA and/or the same assessment method could be used more than once.

In accordance with the original definition of HTA, Table 4 shows that HTAs still predominantly entail synthesizing existing evidence with literature review as the most common method of assessment. This method counts 58 percent of all assessments and is prevalent in 92 percent of HTAs, a fraction that is relatively stable during the whole study period.

Meta-analysis, representing another form of secondary research is used in only 3 percent of the HTAs and accounts for only 2 percent of all assessments.

Surveys account for 7–10 percent of the individual assessments and are present in 15 percent of the HTAs. The prevalence of this method of analysis appears to be relatively stable over the study period. The first RCT included as part of an HTA was carried out in 1999, but more have been observed in latter years. There is also a tendency toward a greater prevalence of evaluations or evaluations based on modeling, ranging from 0 percent in the first time period to 8 percent in the last time period.

Economic evaluations have been applied from the beginning of the study period and account for 14 percent of the individual assessments in the early time period, but the role of economics in HTAs has not increased over the time periods. Rather, the relative frequency of this type of assessment is characterized by a slight negative trend.

The term other method consists of a range of alternative assessment methods and has decreased over the time period from around 20 percent to around 10 percent, signifying greater uniformity in the methods applied in HTA.

DISCUSSION

When considering the results of the present analysis, one has to take into account that the sample is not a perfect sample of the HTA reports produced worldwide. We only included national HTA institutions with a certain level of experience. Furthermore, we only included HTAs written in English or Scandinavian languages, thus excluding reports written in the native language in countries such as The Netherlands and Spain, countries that are relatively active on the international HTA scene.

The results show a significant rise in the number of HTAs during the study period with an almost exponential growth during the last time period. This trend is to be expected, considering that HTA as method first began to diffuse around the world in the late 1980s, and it takes time for new methods to be adopted. The trend is likely to have appeared more pronounced if we had included HTAs from (for example) the regional institutions around the world. Overall, our results disclose an increasing production of HTA reports worldwide, a trend that reflects a greater focus on formalizing input to decision making in the health-care sector.

The time-trends in the application of assessment methods indicate that our results show that literature reviews always have had and still have a prominent role as an HTA method. This picture accords with the original definition of HTA as a method of synthesizing evidence. It is, however, surprising that meta-analysis is so seldom applied as a tool for synthesizing in HTA. One would have expected a rise in the fraction of such assessments following the methodological development of meta-analysis since the late 1980s, but that did not happen.

Assessment methods such as RCT and modeling/evaluation as supplement to other assessment methods are becoming more popular during recent years. This trend suggests that there is an increasing intrest in generating new data when evidence is missing. This trend may be explained by the widening of scope of HTA in the later time periods toward including legal, psychological, ethical, and other aspects in HTAs (6;14;23). Such new questions are likely to raise the demand for more primary research to generate new types of information that cannot be found in the existing literature. Alternatively, the increasing interest in primary research could be a result of supplier-induced demand. The increasing use of external partners as assessors could be introducing a pull toward research associated with higher academic merit.

One may also question the underlying reasons for the general shift toward the use of external partners. A main reason is likely to be that the increasing number of HTAs produced annually gives rise to an increase in demand of qualified analysts and that this demand can only be satisfied by outsourcing some of the tasks. A further explanation could be an increasing need for legitimacy. The results and conclusions presented in HTA reports may have greater legitimacy in the clinical and/or the political arena if these are produced in cooperation with external partners.

Economic evaluations do not constitute an increasing share of assessments. This finding is a surprising trend considering the growth rate in health economics and economic evaluations in general. Economic evaluations were included in less than 20 percent of HTAs. Although it may be warranted to exclude economic considerations in some cases, omitting economic aspects in the majority of HTAs raises questions regarding the applicability of HTAs as input to informed decision-making. As stated by Jonsson (13), “HTA is about a rational health service.” The same point was also made by Maynard and McDaid (17). Rational behavior should involve the realization that new or improved health-care interventions are associated with opportunity costs elsewhere, and the need to quantify these should be apparent.

The category other methods have decreased in fraction used in HTAs during the study period but still entail a relative high fraction of the assessments. One possible explanation for part of this trend could be the slightly higher use of primary research such as RCTs and modeling/evaluation in the late time periods as a substitute for a formal method of synthesis based on existing evidence. Another explanation could be a lesser use of case reports and expert opinion.

The general time trend in national HTAs worldwide is a wider application of HTAs on types of technologies, a more frequent use of external partners as assessors, and most recently a trend toward more frequent use of research methods such as RCTs and modeling/evaluation. Overall, the development in research methods applied in HTAs is relatively small-scaled when considering the degree of development in research methods in general during the past decades. There has been little spillover effect on assessment methods in HTA.

CONTACT INFORMATION

Eva Draborg, PhD (), Assistant Professor, Dorte Gyrd-Hansen, PhD (), Professor, Institute of Public Health–Health Economics, University of Southern Denmark, J.B. Winslowvej 9, DK-5000 Odense C, Denmark

Eva Draborg was financially supported by the Danish Centre for Evaluation and Health Technology Assessment. An earlier version of this paper has been presented at the 1st Annual Meeting of Health Technology Assessment International in Krakow, Poland, June 2004.

References

Banta D. 2003 The development of health technology assessment. Health Policy. 63: 121– 132.Google Scholar
Banta HD, Luce BR. 1993. Health care technology and its assessment. An international perspective. Oxford: Oxford University Press;
Banta HD, Perry S. 1997 A history of ISTAHC. A personal perspective on its first 10 years. Int J Technol Assess Health Care. 13: 430– 453.Google Scholar
Velasco M, Perleth M, Drummoud M, et al. 2002 Best practice in undertaking and reporting health technology assessments. Int J Technol Assess Health Care. 18: 361– 422.Google Scholar
Chan L, Topfer L-A. 2000 Health technology assessment on the Net: A guide to Internet sources of information. Edmonton: Alberta Heritage Foundation for Medical Research & Institute of Health Economics; August.
Draborg E, Gyrd-Hansen D, Poulsen PB, Horder M. An international comparison of the definition and the practical application of health technology assessment. Int J Technol Assess Health Care. 21: 89– 95.
García-Alté A, Ondategui-Parra S, Heumann PJ. 2004 Cross-national comparison of technology assessment processes. Int J Technol Assess Health Care. 20: 300–310.Google Scholar
Goodman CS, Ahn R. 1999 Methodological approaches of health technology assessment. Int J Med Inform. 56: 97– 105.Google Scholar
Hailey D. 2003 Toward transparency in health technology assessment. A checklist for HTA reports. Int J Technol Assess Health Care. 19: 1– 7.Google Scholar
http://www.inahta.or.
http://www.istahc.or.
http://nhscrd.york.ac.u.
Jonsson E. 2002 Development of health technology assessment in Europe. A personal perspective. Int J Technol Assess Health Care. 18: 171– 183.Google Scholar
Jonsson E, Banta D. 1999 Management of health technologies: An international view. BMJ. 319: 1– 3.Google Scholar
Jonsson E, Banta HD, Henshall C, Sampietro-Colom L. 2002 Summary report of the ECHTA/ECAHI Project. European Collaboration for Health Technology Assessment/Assessment of Health Interventions. Int J Technol Assess Health Care. 18: 218– 237.Google Scholar
Liberati A, et al. 1997 EUR-ASSESS Project subgroup report on methodology. Int J Technol Assess Health Care. 13: 186– 219.Google Scholar
Maynard A, McDaid D. 2003 Evaluating health interventions: Exploiting the potential. Health Policy. 63: 215– 226.Google Scholar
Mears R, Taylor R, Littlejohns P, Dillon A. 2000. Review of International Health Technology Assessments (IHTA). Project Report. London: National Institute of Clinical Excellence;
Menon D, Marshall D. 1996; The internationalization of health technology assessment. Int J Technol Assess Health Care. 12: 45– 51.Google Scholar
Menon D, Topfer L-A. 2000 Health technology assessment in Canada. A decade in review. Int J Technol Assess Health Care. 16: 896– 902.Google Scholar
Milne R, Clegg A, Stevens A. 2003 HTA responses and the classic HTA report. J Public Health Med. 25: 102– 106.Google Scholar
Office of Technology Assessment (OTA). 1982 Strategies for medical technology assessment. Washington DC: US Government Printing Office;
Perleth M, Jakubowski E, Busse R. 2001 What is ‘best practice’ in health care? State of the art and perspectives in improving the effectiveness and efficiency of the European health care systems. Health Policy. 56: 235– 250.Google Scholar
Perry S, Gardner E, Thamer M. 1997 The status of health technology assessment worldwide. Results of an international survey. Int J Technol Assess Health Care. 13: 81– 98.Google Scholar
Poulsen PB. 1999. Health technology assessment and diffusion of health technology. Odense: Odense University Press;
Sassi F. 2000 The European way to health technology assessment. Lessons from an evaluation of EUR-ASSESS. Int J Technol Assess Health Care. 16: 282– 290.Google Scholar
Figure 0

Number of Health Technology Assessment Reports by Institution and Country

Figure 1

Number of health technology assessment reports by year of publication

Figure 2

Percentage Distribution of Assessed Technologies by Year of Publicationa

Figure 3

Percentage Distribution of Type of Assessors by Year of Publicationa

Figure 4

Percentage Distribution of Individual Assessments by Year of Publicationa