International comparison of the definition and the practical application of health technology assessment

Eva Draborg; Dorte Gyrd-Hansen; Peter Bo Poulsen; Mogens Horder

doi:10.1017/S0266462305050117

International comparison of the definition and the practical application of health technology assessment

Published online by Cambridge University Press: 02 March 2005

Peter Bo Poulsen and

Eva Draborg: Affiliation:
University of Southern Denmark
Dorte Gyrd-Hansen: Affiliation:
University of Southern Denmark
Peter Bo Poulsen: Affiliation:
Muusmann Research & Consulting AS
Mogens Horder: Affiliation:
University of Southern Denmark and Odense University Hospital

Article contents

Abstract
BACKGROUND
METHODS
RESULTS
DISCUSSION
CONTACT INFORMATION
References

Rights & Permissions

Abstract

Objectives: Health Technology Assessment (HTA) is defined as a policy research approach that examines the short- and long-term social consequences of the application or use of technology. Internationally different institutions have translated this definition to local contexts. In Denmark, HTA is comprehensive with focus on four aspects of the problem in question (technology [clinical evidence], economy, patient, and organization). The objective of this study is to study how the application of HTA differs across leading countries and to study the extent to which Danish HTA reports differ from foreign HTAs.

Methods: A sample of 433 HTA reports published in the period 1989–2002 by eleven leading institutions or agencies in Denmark and eight other countries were reviewed. We looked at the characteristics of the HTA with respect to focus on the four main aspects and the manner in which each aspect has been approached.

Results: The study shows health technology procedures to be the most common type of health technology assessed in HTAs and literature review to be the most often used method of analysis. Policy recommendations are only present in approximately half of the HTA reports.

Conclusions: In the HTAs one generally sees a great focus on the clinical aspect of health technologies, leaving the economic, the patient-related, and the organizational aspect much more unanalyzed. The Danish HTAs generally have a wider scope than HTAs produced in other countries and tend to focus more frequently on patient-related and organizational dimensions.

Keywords

International comparison Content of HTA

Type: GENERAL ESSAYS
Information: International Journal of Technology Assessment in Health Care , Volume 21 , Issue 1 , January 2005 , pp. 89 - 95

DOI: https://doi.org/10.1017/S0266462305050117 [Opens in a new window]
Copyright: © 2005 Cambridge University Press

BACKGROUND

Health Technology Assessment (henceforth, HTA) was originally defined as “a policy research approach that examines the short- and long-term social consequences of the application or use of technology” (13) with main purpose of aiding decision-making in health technology (4). The International Network for Agencies in Health Technology Assessment (INAHTA) has provided the following description of HTA: “technology assessment in health care is a multidisciplinary field of policy analysis. It studies the medical, social, ethical, and economic implications of development, diffusion, and use of health technology” (6). The newly founded society for Health Technology Assessment International (HTAi) defines HTA as “research-based, practice-oriented assessments of relevant available knowledge on the direct and intended consequences of technologies, as well as the indirect and unintended consequences” (5).

Despite or perhaps as a consequence of these rather broad definitions of HTA, some HTA institutions across the world have acquired their own interpretations of HTA to accommodate the local decision-making contexts. Other institutions use the above definition or slightly moderated definitions.

The Danish definition of HTA is formulated as follows: “health technology assessment is a comprehensive systematic evaluation of the assumptions for and consequences of the application of health technology” (9). HTA in the Danish context is viewed in a comprehensive form with focus on four main aspects (technology [clinical evidence], economy, patient, and organization), each of which has several underlying dimensions. Looking at the official definitions of HTA in various countries/institutions may disclose the official and formally stated rationale underlying HTAs, but such formal definitions may differ significantly from the applied state of the art.

The objective of this study is to investigate the locally applied definitions of HTA by using an empirical and explorative approach. Focus will be on the extent to which Danish HTA reports differ from HTAs produced in other countries.

In the literature, no such empirical explorative study of applied definitions of HTAs exists except for the preliminary study preceding this one (15), which only accounted for a smaller fraction of the HTAs in the present sample and were limited to the period 1989 to 1996. The study was done in the early years of HTA and before the now widespread use of HTA as a method for producing evidence for decision making in the health-care sector. Menon and Topfer (11) used the same approach as in the present study on a sample of 117 government-funded Canadian HTAs (national and regional). Perry et al. (14) reported a worldwide study of 103 HTA institutions in twenty-four countries and their application of the HTA concept, including types of technology assessed, content of HTAs, and methods for health technology assessments. Although this study has a worldwide perspective and seeks to disclose the practical application of the HTA concept, it did not review the actual HTA reports but relied on the HTA institutions answers to a questionnaire. Furthermore, it used a very broad definition of HTA institutions, including for example medical societies, for-profit organizations and trade associations. Yet another study used the same survey method but restricted the responders to 50 nonprofit and/or government-financed institutions (10). The subjects of the survey were methodological aspects of HTAs, methods of priority setting, dissemination strategies, and the relationship between HTA and decision making.

METHODS

This study consists of a structured literature review of HTA reports published from leading HTA institutions in the period 1989 to 2002. The sample consists of 433 HTA reports from eleven different HTA institutions in nine countries (Table 1). The sample includes HTAs from institutions that were among the earliest actors on the HTA scene as well as agencies and institutions that were established during the 1990s. The institutions had to be national, nonprofit, and noncommercial institutions, and they were traced from a list of member of International Network of Agencies for Health Technology Assessment (6) and a list from The International Society for Technology Assessment in Health Care (7), which were cross-checked with a list of HTA resources on the Internet (2) and with publications from the Health Technology Assessment Database (8). The following HTA institutions were included in our sample: ASERNIP-S (Australia), AHTAC/MSAC (Australia), CCOHTA (Canada), DACEHTA (Denmark), Health Council (The Netherlands), NZHTA (New Zealand), SMM (Norway), SBU (Sweden), NCCHTA (United Kingdom), AHQR (USA), and VATAP (USA). All of these institutions have several years of experience in the field of HTA and/or have published at least nine HTA reports. These criteria were enforced to secure some level of experience and, therefore, also consistency in definition and application of the HTA concept.

The criteria for HTA reports to be included in the sample were that they had to be full HTA reports. This selection criteria was operationalized by including only reports that the institutions themselves designated as HTA reports, and second, during the review process by further judgment of the content of each report separately. In case of doubt of the character of a certain report, the institution of origin was contacted for advice. In the sample, we consequently excluded all short reports, technology reviews, early warnings, journal articles, and reports concerning methodological HTA issues. The reports, furthermore, had to be written in English or a Scandinavian language and had to be obtainable by the institution's home page or by written contact to the institution.

The process of obtaining the HTA reports was carried out by searching the home pages of the selected HTA institutions. A full list of all HTA reports from each institution was inspected by the first author, and those reports judged to be HTAs were printed from the home pages. In case of HTAs not available by home pages, the institution was contacted by mail or by letter requesting a printed copy of the reports in question.

It should be noted that almost all HTAs that fulfilled the above-mentioned criteria were included in the sample. In some cases, two individually produced reports were interpreted as constituting one single HTA report (for example, when the heading was “Part 1. Technological Review” and “Part 2. Economic Evaluation”).

The review was carried out using a predetermined checklist to collect data on general information and on the selection of analyzed dimensions in the HTA reports. The checklist used for reviewing was divided in two parts. Part one consisted of questions describing the HTA report in general such as country and institution of origin, title, year of publication, name and type of health technology, type of health technology used as the comparator, method of assessment, and type of recommendations. Type of health technology was coded according to the definition used in Banta and Luce (1), where a pharmaceutical is “any chemical or biological substance that may be applied to, ingested by, or injected into humans,” a device is “any physical item, excluding drugs, used in health care”, and a procedure is “a combination of provider skills or abilities with drugs, devices or both.”

Part two of the checklist consisted of questions focusing on the four main aspects of technology assessment as emphasized in the Danish definition of HTA. We chose to focus on a total of twenty-three dimensions across the main aspects. These dimensions are listed in Table 2.

The clinical aspect is measured using six binary questions stating whether the report contains assessment of efficacy, safety, effectiveness, other outcome measures, indications for use of the technology and the relevant target population for that particular health technology (epidemiology). The economic aspect is measured by use of five binary questions stating whether the report contains assessment of efficiency, cost, cost-effectiveness, cost utility, and/or cost benefit. The same procedure was used in relation to the patient aspect, which also contained five questions: assessment of social consequences, ethics, patient acceptance of the technology, psychological implications, and other parameters relevant for the patients. With respect to the organizational aspect, we looked at the following dimensions: diffusion and adoption, centralization or decentralization, utilization, accessibility, demands on the personnel's skills and routines, further education and training of the personnel and finally, other organizational parameters. All of the responses in part 2 were formed as binary responses, yes/no responses, indicating whether or not a certain characteristic was present in the individual report.

The present analysis is split up in two subanalysis. The first has an explorative approach and focuses on providing a descriptive analysis of the definition and content of HTA reports. In the second subanalysis, the individual dimension is given a score of “1” when the dimensions is analyzed in the report and a score of “0” when it has not been dealt with in a given HTA report. The scores are then summed up for each aspect and for the individual HTA report as a whole, and institution/agency-specific mean scores are calculated. The score is based on the assumption of equal weighting of all dimensions, and the score is used as an indicator of the how broadly focused the average HTA is in that particular institution/agency. The higher the mean score an HTA institution attains, the wider the scope of applied HTA in that agency/institution.

RESULTS

Figure 1 illustrates the time distribution of HTA reports in the sample and shows that 52 percent of the reports are published in the period 2000–2002, which means that almost the same share of HTAs in the sample are published in the past 3 years as in the first 11 years of the study period. This development reflects an increase in the number of institutions producing HTA reports and in the number of produced reports by institution.

Number of health technology assessment reports by year.

Table 3 illustrates selected characteristics of the reports, and it shows an almost even distribution between HTA reports made by the institutions themselves and commissioned HTA reports. The number exceeding 100 percent means that some HTA projects are made in collaboration between the institutions' own personnel and persons external to the institution.

The type of assessed technology is primarily health technology procedures, which account for approximately 60 percent of the technologies that are assessed, and/or used as a comparator in the HTA. Approximately one third of all HTA reports have no comparator. The explanation behind this finding is that the use of placebo was defined as a “none comparator.”

More than 90 percent of the HTAs use literature review as a method of assessment. Primary research methods such as randomized control trials and surveys are used less often (less than 20 percent), and other methods (such as the inclusion of expert opinions) are applied in 20 percent of the HTA reports.

Table 3 shows that policy recommendations are present in only half of the HTA reports. A similar proportion of HTAs focus their recommendations on future research. In as many as 33 percent of the HTAs no explicit recommendation is stated. Table 2 shows the results of recording the dimensions and main aspects assessed in the HTA reports.

The overall results show that the clinical aspect is the most frequently included aspect, followed by the economic aspect. The patient-related aspect and the organization aspect are less often incorporated in HTAs.

The clinical aspect differs from the three other aspects, because the majority (5 of 6) of clinical dimensions are included in more than half of the HTA reports. Frequently (in more than 50 percent of HTAs) only one of the economic dimensions is included. A similar fraction of HTA reports do not discuss any issues that are patient related nor do they draw organizational dimensions into the assessment. The overall picture is that there is greatest focus on the clinical aspect in the sample.

Looking closer at the specific dimensions effectiveness is assessed more often than efficacy and indications is the most frequently addressed clinical dimension. Economic dimensions were most often assessed by way of a cost analysis, and if a full economic evaluation was performed, it was most often in the form of a cost-effectiveness analysis. Organizational issues addressed were primarily health personnel skills and routines and demand for training and/or education. Patient-related issues were evenly distributed across a series of dimensions but with a tendency to focus on acceptability and psychological reactions.

In addition to the aggregate overview, we also analyzed the scope of the HTA reports at a more detailed level. As explained in the methods section, the individual counting of dimensions assessed in the HTA reports were used for calculating HTA scores for the HTA reports individually. Each dimension included in a report initiated one score point. Hence, the maximum score a single report could receive would be twenty-three points (equivalent to the number of dimensions). A high score indicate that more dimensions and/or aspects are assessed in the HTA reports. The procedure resulted in five different HTA scores per HTA; four for the main aspects and a total score. We report the mean scores per HTA institution in Table 4.

The Danish HTAs obtain a higher mean score than the remaining institutions. This indicates that Danish HTA reports generally have a greater scope and focus on a larger set of issues than other institutions. Swedish HTAs also tend to have a broad scope, whereas the lowest total HTA score is registered for AHQR, USA.

Although one should be careful in comparing the absolute scores across the four main aspects (note that each main aspect does not have the same number of dimensions), the statistics reiterate the earlier result that the clinical aspects are treated much more thoroughly in HTA reports.

Organizational, economic, and patient aspects are frequently not incorporated, and if they are, only a limited number of questions are answered. Economic issues are more frequently represented than are patient issues. This order of priority seems fairly consistent for all institutions in the sample.

It is worth noting that the high Danish score is not explained by a higher score on the clinical aspect. The difference between the Danish scores and the other institutions score are most distinct when looking at the patient score and the organizational score, indicating that DACHETA more often includes patient-related dimensions and organizational dimensions in their health technology assessments compared with the other ten HTA institutions.

DISCUSSION

There are some limitations to the present study. First, the study does not include an exhaustive set of HTA institutions. We have chosen to focus on those institutions that have significant experience with HTA to analyze how these institutions choose to apply the concept of HTA. Second, HTA reports that are not written in English or Scandinavian languages are excluded from the analysis. This strategy is clearly of disadvantage to non-English and non-Scandinavian-speaking countries. The Netherlands, for example, have produced many reports written in Dutch. However, the sample of HTA reports written in English is assumed to be identical in nature to those written in Dutch; hence, this bias is assumed not to affect our conclusions.

Also, when calculating total HTA scores (in Table 4), one may question the appropriateness of giving each dimension an equal weight. One could naturally argue that some dimensions are of greater importance than others. We do not seek to judge the relative quality of HTAs on the basis of this HTA score. Rather, the score results should merely be interpreted as signaling the scope of the HTA in question.

Figure 1 shows that the number of HTA reports performed over the years has exploded, especially in the period 1997 to 2002. This increase can be seen as reflecting a significant dissemination of HTA as a method.

A striking result is that more than 90 percent of the HTAs use literature review as method of assessment, which is in accordance with the original aim of HTA as method of synthesizing existing evidence for the purpose of decision making not primarily producing new evidence (3;12). The types of technologies in focus in HTAs are primarily procedures and procedures are very often also used as comparators. It is noteworthy that as much as one third of the HTAs do not describe the comparator in their assessment. Performing an assessment of an intervention without detailed consideration of the characteristics of the alternative strategy must be deemed inappropriate as a basis for policy making in most instances; hence, the usefulness of such assessments is surely limited.

Policy recommendations are frequently omitted from HTA reports. Whether HTA reports represent a good aid for decision making when such synthesis is omitted should be focus of future research.

The overall result of the present study shows that there is generally a great focus on the clinical aspect of a HTA and that holds for all the HTA institutions in the sample. The economic aspect is seemingly judged as having greater importance than the organizational issues and the patient-related aspects. This relative weighting seems fairly consistent across the institutions in the sample.

HTAs originating from DACEHTA (Denmark) have the widest scope relative to the 10 other HTA institutions, and this finding is mostly due to inclusion of dimensions related to the patients and related to the organization. The reason for this wider scope is that in Denmark HTA consistently and systematically has been presented as a method of evaluation that should focus on all four aspects (clinical evidence, economy, patient, and organization). Theoretically, this approach to HTA could result in a better foundation for a subsequent decision-making process, when facing complex policy issues.

However, the present study has not had the purpose of investigating what information was needed in the specific decision-making situations that the HTA reports targeted for. A topic for future research, therefore, could be to analyze whether characteristics of HTAs affects the influence that such reports have on political decision making.

The overall conclusion that can be drawn from our study is that the popularity of HTA is increasing, but we must also note that the definition of HTA as a multidisciplinary field of policy analysis that studies the medical, social, ethical, and economic consequences of health-care interventions does not hold. In practice, HTAs are frequently more narrowly defined.

CONTACT INFORMATION

Eva Draborg, PhD, Assistant Professor (eud@sam.sdu.dk), Dorte Gyrd-Hansen, PhD, Professor (dgh@sam.sdu.dk), Institute of Public Health–Health Economics, University of Southern Denmark, Winslowparken 19.3, DK-5000 Odense C, Denmark

Peter Bo Poulsen, MSc, PhD, Manager of Evaluation and Analysis (pbp@muusmann-as.dk), Muusmann Research & Consulting AS, Haderslevvej 36, DK-6000 Kolding, Denmark

Mogens Horder, DM Sci, Dean, Professor (MHorder@health.sdu.dk), Faculty of Health, University of Southern Denmark, Winslowparken 17.1; Head, Department of Clinical Biochemistry, Odense University Hospital, Sdr. Boulevard 29, DK-5000 Odense C, Denmark

Dr. Draborg is financially supported by the Danish Centre for Evaluation and Health Technology Assessment. An earlier version of this paper has been presented at the 19th Annual Meeting of The International Society of Technology Assessment in Health Care in Canmore, Canada, June 2003.

References

Banta HD, Luce BR. 1993. Health care technology and its assessment. An international perspective. Oxford: Oxford University Press;

Chan L, Topfer L-A. Health technology assessment on the Net: A guide to Internet sources of information. Edmonton: Alberta Heritage Foundation for Medical Research & Institute of Health Economic; August 2000.

EUR-ASSESS Project Subgroup on Methodology. 1997; Methodology guidance for the conduct of health technology assessment. Int J Technol Assess Health Care. 13: 186– 219.

Goodman CS. 1998; Healthcare technology assessment: Methods, framework, and role in policy making. Am J Manag Care. 4: (Special Issue) SP200– SP214.Google Scholar

Health Technology Assessment International. Available at http://www.htai.org.

International Network for Agencies in Health Technology Assessment. Available at: http://www.inahta.org.

International Society for Technology Assessment in Health Care. Available at: http://www.istahc.org.

Health Technology Assessment Database. Available at: http://nhscrd.york.ac.uk.

Jørgensen T, Hvenegaard A, Kristensen FB. 2000; Health technology assessment in Denmark. Int J Technol Assess Health Care. 16: 347– 381.Google Scholar

Mears R, Taylor R, Littlejohns P, Dillon A. 2000. Review of international health technology assessments (IHTA). Project report. London: National Institute of Clinical Excellence;

Menon D, Topfer L-A. 2000; Health technology assessment in Canada. A decade in review. Int J Technol Assess Health Care. 16: 896– 902.Google Scholar

National Institute of Health. 2003. Etext on health technology assessment (HTA) information resources. Bethesda, MD: National Institute of Health;

Office of Technology Assessment (OTA). 1976. Development of medical technology: Opportunities for assessment. Washington DC: US Government Printing Office;

Perry S, Gardner E, Thamer M. 1997; The status of health technology assessment worldwide. Results of an International Survey. Int J Technol Assess Health Care. 13: 81– 98.Google Scholar

Poulsen PB. 1999. Health technology assessment and diffusion of health technology. Odense: Odense University Press;