A recent development in Europe, North America, and Australasia is that Health Technology Assessments (HTA) are more often being “hard wired” into resource allocation decisions, such as those about the reimbursement of drugs and other health technologies. Given this shift in emphasis, a set of key principles (KP) has been proposed for the improved conduct of HTA for resource allocation decisions (Reference Drummond, Schwartz and Jonsson8).
Since their publication, the principles have generated a considerable amount of debate. Some have questioned whether the principles are realistic and whether it is fair to expect HTA organizations to satisfy all of them. Others have pointed out that, among the principles, there is a general trade-off between rigor and inclusiveness on the one hand and timeliness on the other (Reference Banta4;Reference Drummond, Schwartz and Jonsson7;Reference Hailey10;Reference Neuhauser14). A detailed description of the principles and the rationale for each is provided elsewhere (Reference Drummond, Schwartz and Jonsson8). Table 2 enumerates the principles, divided into four sections: structure of HTA programs, methods of HTA, processes for conducting HTA, and the use of HTA findings in decision making.
Latin America (LA) is a heterogeneous region with wide ethnic, cultural, and socioeconomic diversity (15). As opposed to Europe, Australia, or Canada, most countries in LA have fragmented healthcare systems in which different subsystems coexists, with public, social insurance and private insurance, each having a different decision making process. Nevertheless, LA is a region with increasing use and influence of HTA in decision making (Reference Banta and Almeida3;Reference Banta5;6;Reference Gomez-Dantes and Frenk9;Reference Iglesias, Drummond and Rovira11;Reference Rubinstein, Pichon-Riviere and Augustovski17) with HTA agencies affiliated to INAHTA (International Network of HTA agencies) in four countries: Argentina, Brazil, Chile, and Mexico (12). In addition, HTA was formally used to shape benefit packages in Argentina, Uruguay, and Chile; and countries like Brazil, Mexico, and Colombia have a formal fourth hurdle system in place that require evaluation of new technologies using HTAs (Reference Augustovski, Bastardo and Caso2).
The study of Drummond and colleagues (Reference Drummond, Schwartz and Jonsson8), although global in scope, was based mainly in the experience of countries with advanced HTA programs, No study, to our knowledge, formally evaluated how relevant or applicable HTA users or doers consider these KP. Thus, focusing specifically in our geographical area of work, we undertook a study to address these concerns in Latin America.
OBJECTIVES
To explore whether these KPs are relevant and useful and to what extent they are being currently applied in Latin America, a region with more resource constrained countries, fragmented healthcare systems and where HTA is being increasingly used for resource allocation decisions.
METHODS
We performed a confidential, self-administered, Web-based survey using KMail Software® V 4.6.5. For each of the 15 KPs, the survey included a brief description of the principle followed by items that explored two dimensions: (i) how relevant (useful) the respondent considered the principle was in his or her country, and (ii) to what extent the respondent considered that this principle was being applied in his or her country. Both items had a 1 to 10 scale, with anchors at 1 (no relevancy/usefulness, or never applied) and 10 (extreme relevancy/usefulness or always applied).
Additional data retrieved included information about the respondent and the kind of decisions for which the HTA reports were used. The survey was administered in neutral Spanish and was piloted with potential respondents to ensure technical functioning, relevance and understanding of the questions.
In addition, we computed the “gap” between each key principle´s relevance score and its degree of current application. The higher the gap, for a maximum possible score of 9 points, the bigger the mismatch between the perceived importance of the KP and its real application.
An e-mail with a brief explanation about the project and the link to the Web site with the complete survey was sent to 11,792 HTA researchers and HTA research users (9,426 in Argentina and 2,366 in other LA countries). Respondents were identified by means of our own network of researchers, managers, policy makers, and other research users that are registered at IECS (Institute for Clinical Effectiveness and Health Policy) Web page. Registration is free, but mandatory to access IECS's HTA documents. The mailing also included national health ministries, HTA agencies, university/research organizations, third party payers, medical device and pharmaceutical industries, regulatory authorities, and Pan American Health Organization Listserv on health technologies.
Descriptive statistics were computed for all variables measured. We used the chi-square test to determine differences in categorical variables, and the independent-sample t-test or Wilcoxon-Rank Sum Test for continuous variables. The p values of less than 0.05 (two-tailed) were considered statistically significant. Data analyses were performed using Stata (version 8.0).
RESULTS
Between June 16 and September 4 of 2009, we received 1,142 responses with a global response rate of 9.7 percent. Sixty-six responses were eliminated as they belonged to non-LA countries. The characteristics of the respondents are summarized in Table 1. Fifty-five percent of the surveys corresponded to Argentina, and 45.3 percent were from 18 other LA countries, mainly Peru (7.4 percent), Brazil (6.0 percent), Colombia (6.0 percent), Uruguay (4.8 percent), Mexico (3.8 percent), and Chile (2.5 percent). Fourteen percent of the respondents defined themselves as HTA researchers and 86 percent as HTA users. Around one third stated that they use the HTA reports at an institutional level for decisions related to coverage and reimbursement of health technologies, one third used them at institutional level for other decisions not related directly to coverage (e.g., Clinical Practice Guideline development), and another third used them for clinical decisions at the patient level. The majority of the respondents belonged to the public and private health sector, followed by the academic and government sectors.
Table 1. Respondents´ Characteristics
LA, Latin America; HS, Health Sector; Pharm, Pharmaceutical.
All fifteen KPs were considered relevant and received high scores in the relevance/usefulness scale with mean scores that ranged from 8.3 to 9.2. Results are summarized in Table 2. As expected, they also received low scores in the ratings of current application, means ranged from 3.2 to 4.9. All KPs showed intermediate to high “gaps” (mismatch between its importance and its real application) that ranged from a mean of 4.3 to 5.7.
Table 2. Mean Scores in the Relevance, Application, and Gap Scales by KP
HTA, Health Technology Assessment; SD, standard deviation
The subgroup of KPs related to Methods and to the Use of HTA in decision making received the higher scores in the relevance scale (with mean scores of 9.00 and 8.94, respectively), followed by those related to Processes for conducting HTA (8.58) and Structure of HTA programs (8.52). The principle that was considered most relevant was KP number 5: HTA Should Incorporate Appropriate Methods for Assessing Costs and Benefits, followed by principles 15 and 13: The Link Between HTA Findings and Decision-Making Processes Needs to Be Transparent and Clearly Defined, and HTA Should Be Timely.
Regarding the question that explored to what extent the respondent considered that each principle was being applied in his or her country, the subgroup of KPs related to the Processes for conducting HTA and to the Use of HTA in decision making received the lowest scores (3.35 and 3.39, respectively). The group of KPs related to Methods received the highest score (4.02), meaning a higher level of perceived application.
The gap between the relevance of the principle and the degree of current application can highlight those areas where more urgent attention is needed in the region. The higher gaps were observed in the group of principles related to the Use of HTA in decision making (mean score 5.32) and to the Processes for conducting HTA (mean score 5.26). The principles that presented the higher gap were principle 12: The Implementation of HTA Findings Needs to Be Monitored, 7: A Full Societal Perspective Should Be Considered When Undertaking HTAs and 15: The Link Between HTA Findings and Decision-Making Processes Needs to Be Transparent and Clearly Defined.
Differences in the mean scores across KPs were generally small, although statistically significant in many cases. Despite this, and more importantly, the KP order of relevance was homogeneous among countries. KPs related to Methods and to the Use of HTA in decision making were uniformly considered the most relevant in all countries. Similarly, the greater level of current application was identified in KPs related to Methods and to Structure of HTA programs; and the larger gaps were observed in those related to Processes for conducting HTA and to the Use of HTA in decision making.
As more than half of our sample was composed of Argentinean subjects, we considered important to evaluate whether significant differences existed between Argentina and the other countries. Although some differences in mean values were observed (i.e., responses from Argentina have lower relevance scores, lower scores in the application scales and higher gaps) the KP order of relevance was similar to the other countries.
Higher level of development of HTA could be expected in countries with INAHTA agencies (Argentina, Brazil, Chile, and Mexico). In effect, responses from these countries showed higher values in the application scores and lower gaps, mainly in the KPs related to methods (Gap 5.34 versus 4.81, p < .01).
Responses from countries with health economic evaluation (HEE) Guidelines (Mexico and Brazil) had higher scores in the relevance scale as compared to other countries (9.16 versus 8.64, p < .01), higher scores in the degree of current application scale (5.3 versus 3.4, p < .01), and lower gaps (3.84 versus 5.21, p < .01). As is shown in Table 3, these results were consistent and statistically significant across all the KP subgroups.
Table 3. Mean Scores of KP Subgroups: Countries With and Without National Methodological Guidelines for the Conduct of Health Economic Evaluations
Note. No HEE guideline: Countries without national Methodological Guidelines for the conduct of Health Economic Evaluations (HEE) – n = 971. With HEE guideline: Countries with national Methodological Guidelines for the conduct of Health Economic Evaluations (HEE) – n = 105 (Brazil and Mexico).
KP, key principle.
Researchers, as compared to research users, tend to consider KPs more relevant (Table 4). This was observed mainly in KPs related to the Use of HTA in decision making (9.22 versus 8.89, p < .01) and in Methods (9.26 versus 8.97, p < .01). However, researchers were also less critical when judging the level of current application and scored higher in the application scale in all the four subgroups.
Table 4. Mean Scores of KP Subgroups: HTA Researchers vs. HTA Research Users
KP, key principle; Research., researcher; HTA, health technology assessment.
DISCUSSION
The importance of guidelines to frame the conduct of HTA has paralleled the growth of HTA and HEE to inform decisions on resource allocation in most of the developed countries, especially in Europe, Canada, and Australia (Reference Liberati, Sheldon and Banta13;Reference Sorenson, Drummond and Kanavos18–20). In this regard, the paper by Drummond et al. about setting KPs to improve the conduct of HTA for resource allocation decisions (Reference Drummond, Schwartz and Jonsson8) adds important methodological as well as implementation recommendations to help improve the value of HTA to inform this process. Although the key principles were formulated to be broadly relevant, little is known to our knowledge about current application and relevance of these different KPs in developing countries, and particularly in Latin America. Most countries in our region have major healthcare problems related to both equity and efficiency, in a context of pluralistic and fragmented healthcare systems. Taking this into account, it is more difficult to assume that all KPs that were formulated mainly for developed countries, most of them with single payers or national health systems, have the same relevance or applicability in developing countries.
Some of the findings of our study bring relevant information into this debate. LA respondents considered the KPs related to Methods and to the Use of HTA in decision making to be more relevant. These KPs might be thought of being more “generalisable and universal,” meaning that, they are “insensitive” to the healthcare system. That is, regardless of the characteristics of healthcare systems, ‘HTA should be conducted appropriately’ and ‘the results should be applied in a timely and transparent way’ no matter how great the differences are the settings and contexts where HTA structure and process take place. Of interest, higher scores in the level of current application were also observed in KPs related to methods, reflecting perhaps an acceptable level of development of HTA research capacity in the region. This could be the case in some countries like Argentina, Brazil, Chile, Mexico, and recently Colombia, where the HTA infrastructure of some of their universities and academic centers, and also governmental agencies in Brazil and Mexico, is rapidly growing. On the other hand, the level of current application of KPs related to methods was much lower in countries with less development of institutional bodies focused on HTA (for example, those without INAHTA agencies).
The most interesting findings of our study came from the analysis of the gaps between how the respondents scored the relevance and the corresponding level of application of each KP in their settings. These gaps were generally expected, as HTA is poorly developed in the LA region. However, they were not evenly distributed. Decision makers consistently perceived the gaps as larger than those perceived by researchers across all KPs. This may be explained by the higher pressure that decision makers have and their demand for bigger leaps needed in the HTA field. As expected, we also found that gaps were larger in countries with lower development of HTA.
Wider gaps might well also be a reflection of the greater demands placed on researchers and decision makers by some of the principles. Not surprisingly, the bigger gaps were observed in KPs related to the use of HTA in decision making, followed by those related to processes for conducting HTA. These important results certainly address a major challenge for all the countries in the region regarding the use and application of HTA and economic evidence for policy making. Even in countries with HTA agencies and quite developed HTA programs (Argentina, Brazil, Chile, and Mexico), the link between HTA and decision making is not clearly defined (Reference Augustovski, Bastardo and Caso2;Reference Banta and Almeida3;Reference Banta5;Reference Gomez-Dantes and Frenk9;Reference Rubinstein, Pichon-Riviere and Augustovski17). These results could also reflect that the “macro” level of HTA (health system) still lag behind the “meso” or “micro” level (research capacity) even in the most advanced countries in the region with respect to HTA.
Even though the key principles seem to be important to the application of HTA for resource allocation or priority setting, it is not clear whether this list is exhaustive enough or “transculturally appropriate” if taken as a package in low or middle income countries, considering their more limited resources and their different socio-political and healthcare context. On the other hand, as Banta and Almeida and Hailey commented, these principles could be considered rather idealistic or unachievable (Reference Banta4;Reference Hailey10).
Our study has some limitations. (i) Low response rate is a common feature to most Web-based surveys. (ii) A low dispersion in scores (ceiling effect). We anticipated that something like this could happen. For this reason, we avoided the use of a Likert-type scale, as probably all the principles would had been classified as ”very important” and the ceiling effect would have been even more severe. Despite this, the differences in the mean scores across KPs, although statistically significant in many cases, showed little variation. Therefore, in interpreting the results we consider the KPs’ order of relevance to be more important than the mean scores. (iii) The majority of responses were from one country (Argentina); however, the main results remained unchanged in subgroup analyses by country. (iv) The interpretation of “level of current application” may have varied between respondents (e.g., across HTA researchers within a single agency or across agencies). (v) The survey design was suboptimal and the metric to show each KP gap has not been validated. (vi) Finally, the respondents’ interpretation of KPs could have been heterogeneous, even though a brief description of the principles was included in the survey along with reference to the original paper. Understanding Spanish could have also been a problem for Portuguese-speaking respondents in Brazil.
CONCLUSIONS
Although economic considerations to prioritize resource allocation decisions are increasingly being accepted in Latin America, the use and application of HEE or HTAs is still very limited. Lack of conceptual and technical knowledge, difficulties in accessing studies, lack of credibility in data sources and in their external validity, together with institutional fragmentation in health sector are barriers to disseminating HTA or HEE (Reference Augustovski, Iglesias, Manca, Drummond, Rubinstein and Martí1–Reference Banta and Almeida3;Reference Banta5;Reference Gomez-Dantes and Frenk9;Reference Rubinstein, Belizan and Discacciati16;Reference Rubinstein, Pichon-Riviere and Augustovski17). Our analysis shows that these generic KPs seem to be very relevant to most HTA researchers and users in LA. However, the perceived level of current application in resource allocation decisions was considered uniformly poor. In the original paper on KPs, authors highlighted: “. . .Our goal in formulating the proposed principles was to stimulate the discussion of HTA programs goals and procedures to enhance the rigor, validity, and usefulness of HTA. . .” (Reference Drummond, Schwartz and Jonsson8). In the case of LA and other low or middle income countries, our goal should be to promote a policy dialogue among researchers and policy makers about HTA as an extremely valuable tool to inform resource allocation decisions. Our study can help to focus the analysis and the debate on KPs which were found to have larger gaps, especially those related to the link between HTA and decision making.
CONTACT INFORMATION
Andres Pichon-Riviere, MD, MSc, PhD (apichon@iecs.org.ar), Professor, School of Public Health, University of Buenos Aires, Marcelo T de Alvear 2202; Executive Director and Director of Health Technology Assessment and Economic Evaluations Department, Institute for Clinical Effectiveness and Health Policy (IECS), Viamonte 2146 – 3rd floor, Buenos Aires, 1114, Argentina
Frederico Augustovski, MD, MSc (faugustovski@iecs.org.ar), Director, Economic Evaluation & HTA, Institute for Clinical Effectiveness and Health Policy (IECS), Viamonte 2146, 3er piso, (C1056ABH) Buenos Aires, Argentina; Staff Physician, Department of Family and Community Medicine, Hospital Italiano de Buenos Aires, Gascón 450, Buenos Aires, C1181ACH, Argentina
Adolfo Rubinstein, MD, MSc, PhD (arubinstein@iecs.org.ar), Professor, School of Public Health, University of Buenos Aires, Marcelo T de Alvear 2202; President, Institute for Clinical Effectiveness and Health Policy (IECS), Viamonte 2146 – 3rd floor, Buenos Aires (1114), Argentina
Sebastián García Martí, MD (sgarciamarti@iecs.org.ar), HTA Group Executive Coordinator, HTA and EE Department, Institute for Clinical Effectiveness and Health Policy (IECS), Viamonte 2146 3er piso, Buenos Aires, C1056ABH, Argentina
Sean D. Sullivan, BSPharm, PhD (sdsull@uw.edu), Professor and Director, Pharmaceutical outcomes Research and Policy Program, University of Washington, 1959 NE Pacific Avenue, Seattle, WA 98195
Michael F. Drummond, PhD (md18@york.ac.uk), Professor of Health Economics, Centre for Health Economics, University of York, Alcuin A Block, Heslington, York, North Yorks, YO 105D, United Kingdom
CONFLICT OF INTEREST
A. Pichon-Riviere received partial funding for this work from a Global Health Leadership Award from Global Health Research Initiative (a partnership of the Canadian International Development Agency, the Canadian Institutes for Health Research, Health Canada, and the International Development Research Centre). M. Drummond and S. Sullivan were authors of the original instrument but had no role in the collection and analysis of the data. The other authors report having no potential conflicts of interest.
