As the number of new healthcare interventions grows and the volume of new knowledge from clinical research increases, making judgments on available technology is no longer a simple task for healthcare professionals. Health technology assessment (HTA) fills this need by bridging research and health decision-making processes (Reference Battista and Hodge3). It helps to meet two conflicting goals of the healthcare system—to provide the best available treatment to patients and to enhance the optimal allocation of limited resources (Reference Scott and Moskowitz22). Over the years, the importance of the role of HTA increased, resulting to the establishment of HTA agencies in many countries.
In 2005, an ordinance of the Ministry of Health (MoH) established the Agency for Health Technology Assessment in Poland (AHTAPol), whose main objective was to provide the Ministry of Health with reimbursement recommendations (17). Beginning as a unit under the Ministry, the AHTAPol is now its own legal entity with its own budget, operating at the national level under the supervision of the Ministry of Health. Its important contribution to the decision-making process granting authorization and diffusion of health technologies in Poland has recently been emphasized with the amendment of the law on healthcare services being financed by public funds in August 2009 (16).
Two years after its operation, AHTAPoL published the first Polish HTA guidelines covering applications for reimbursement of new chemical entities (NCE) (1). Three types of evidence were required in the HTA dossier: clinical effectiveness, economic analysis, and analysis of the impact of the NCE on the healthcare system. The guidelines highlighted the vital roles of HTA doers and HTA users. HTA doers (manufacturers of a healthcare technology) are responsible for the delivery of relevant scientific information into the process of developing reimbursement recommendations while HTA users (AHTAPoL) must ensure the transparency and high quality of the process.
The first stage in the development of any HTA recommendation starts with a request of the Ministry of Health to AHTAPol. This request can be a result of a submission of an application for reimbursement with the accompanying HTA dossier by a sponsor to the Ministry of Health. During the review at AHTAPol, the HTA dossier is checked for compliance to AHTAPoL guidelines. Subsequently, AHTAPol prepares an assessment report that summarizes the evidence for the intervention of interest with help from external experts and reviewers. The report includes (i) the description of the technology and its comparators; (ii) analysis of its reimbursement status in other jurisdictions; (iii) analysis of clinical guidelines; (iv) review of scientific evidence on clinical efficacy, safety, cost- effectiveness, cost-utility, and budget impact (these may be taken from the application dossier or collected by the AHTAPol from secondary sources); (v) cost of the drug; and (vi) condition for the public financing options of the given technology (16). Then the assessment report is submitted to the Consultative Council, which is an independent advisory body consisting of external experts invited by the MoH. The Consultative Council deliberates and prepares the recommendation on the inclusion of the drug in the National Formulary and target beneficiaries of the decision such as specific groups of patients or therapeutic programs. The HTA recommendation is relayed to the president of AHTAPol, who in turn, endorses it to the Minister of Health. The final recommendation and its accompanying documents are published on the AHTAPol's Web site. As stipulated by the law, the HTA process from receipt of the application to the final recommendation must take only 45–60 days (16).
There are global initiatives from the International Network of Agencies for Health Technology Assessment (INAHTA) and the European Collaboration for Health Technology Assessment Project (ECHTA) that outline best practices in undertaking, generating, reporting, appraising, and evaluating HTA recommendations. These come in the form of methods toolkits, standard procedures, and checklists (Reference Busse, Orvain, Velasco and Perleth6;Reference Hailey11). It was envisioned that these international guidelines will serve as tools for developing HTA skills in different environments, inform HTA doers and HTA users, as well as provide benchmarks for improvement in performance. They can also be a tool for identifying aspects of the HTA process that are given emphasis at a particular setting, fostering a greater understanding of how a particular HTA agency works.
The initial operating years of any HTA agency can be challenging. There is a need to continually inform its performance and improve its processes. Furthermore, it is important to adopt best practices to ensure the validity of any HTA recommendation and its acceptance in the local environment as well as in the international setting. Following a common body of principles and methods can greatly reduce heterogeneity, enhance credibility in providing the best available evidence on a wide range of medical interventions and facilitate high quality of resource allocation decision and international comparability (Reference Hilman12).
By nature HTA can be complicated and potentially controversial (Reference Drummond, Schwartz and Jönsson8). There are many stakeholders interested in the HTA process and results. It is thus mandatory that transparency be observed. The details of the assessment and discussions leading to the recommendation must be clearly stated, freely available, and accessible for all. Moreover, decisions must be based on rigorous scientific evidence to prevent distortions in health resource allocation decisions. This makes quality critical to the process. Quality is fostered when appropriate methods are undertaken and described accordingly and when all relevant outcomes of interest and its consequences are included.
This study was undertaken to answer the need to evaluate the HTA process in Poland—to define its accomplishments and to inform its implementation. There is much to be learned about a HTA agency's performance from publicly available HTA appraisal reports (Reference Lafortune, Farand and Mondou14). The primary objective of this study is to determine the extent to which transparency and quality, as defined by international standards, are met in the HTA process in Poland. A secondary objective is to describe the outcomes of the HTA process and factors that are associated with these outcomes based on the Polish experience. It is an aspiration that the conclusions drawn from this study will provide HTA doers and HTA users with valuable recommendations for further improvement of the HTA process in Poland. To our knowledge, this is the first time that such an endeavor is done locally, if not internationally, as most evaluations focused more on the scientific merits of the evidence submitted and not on aspects of the appraisal process itself.
METHODOLOGY
All published HTA reports, consisting of the HTA recommendation (appraisal decision) and Consultative Council meeting proceedings, were examined. Available HTA reports of drug technologies issued in 2008 were downloaded from the AHTAPol Web site on May 10, 2009, and revisited on August 25, 2009. Only full HTA reports with complete recommendations and meeting proceedings were considered.
HTA Score Card
An HTA score card was specially developed for this study under the following assumptions: the appraisal process entails critical evaluation of all consequences of the implementation of a new health technology into the clinical practice, thorough analyses in the assessment and application of judgment on relevant HTA aspects (19); the standards of transparency and quality must be equally applied to HTA reports and their appraisal; and because Poland is a member of INAHTA and ECHTA, current standards of these organizations will apply. The HTA score card contained the main domains of the INAHTA and ECHTA checklists relating to transparency and quality. Key guide questions under each domain were identified using all available international HTA checklists.
The INAHTA checklist had five domains while the ECHTA checklist had twelve domains. All seventeen domains were adopted in the score card (nine items for transparency and eight items for quality). The various items included in the HTA score card, their corresponding domains in the INAHTA and ECHTA checklists and detailed descriptions are shown in Table 1.
Table 1. HTA Score Card: Key Items and Guide Questions
Using guide questions for each key item, two independent evaluators analyzed and scored each HTA report. Any disagreement was settled by consensus. If an aspect in the HTA score card exists or is covered in the documents being reviewed, a score of 1 is given, otherwise 0. A score of 0 assumed that the aspect was lacking in the submission documents or was not appraised during the deliberations. All items were equally weighted in deriving a summary score.
Safety, efficacy, and economic aspects of a given health technology are considered as key factors in the appraisal process. Therefore, HTA reports were scrutinized to note any comments or issues raised at the Consultative Council's meeting. Comments on efficacy and safety were classified whether they were related to selection of data and data synthesis (Figure 1). For the economic analyses, comments on the assumptions and methodology were noted (Figure 2).
Figure 1. The Consultative Council's comments regarding efficacy of appraised drug technologies in 2008.
Figure 2. The Consultative Council's comments regarding safety of appraised drug technologies in 2008.
An advice to reimburse a given technology is named a positive recommendation while a negative recommendation proposes that financing of the drug/medical device from public funds is not advised. Conditional recommendations (“finance temporarily,” “finance provided that some precisely criteria are met in particular indications,” “finance provided that a cost-effective way of financing was assured”) were considered part of a positive recommendation and were not analyzed separately by the authors of the article (2).
Descriptive analyses of the scores of all included HTA reports were done. Data were stored and analyzed using Microsoft Excel 2000 for Windows.
RESULTS
There were seventy-five HTA recommendations issued by AHTAPol in 2008, of which three were non-pharmacologic submissions and another three did not have complete documents online. After exclusion of these, a total of sixty-nine HTA appraisal reports, including Consultative Council meeting proceedings and eventual HTA recommendation were included in the study. A description of the different organ systems covered by the HTA reports is given in Table 2.
Table 2. Distribution of Submitted HTA Drug Appraisal Reports by Organ System (2008)
From the sixty-nine reviewed appraisals, thirty-six (52 percent) received positive recommendations while thirty-three (48 percent) had negative recommendations. A negative recommendation was usually given when insufficient evidence, incorrect estimation of the target population or questionable data quality was provided for the safety, efficacy, cost-effectiveness, and impact to the healthcare budget criteria of the assessments. In many instances, several comments were raised simultaneously.
Evaluation of HTA Appraisal Reports
Transparency. Of the sixty-nine reports reviewed, contact details of authors were provided by sixty-seven reports (97 percent) and conflicts of interest were declared in sixty-six appraisals (96 percent). The purpose, context, and scope of the health technology appraisal were specified in all the reports (100 percent). The majority of reports stated that the HTA was commissioned and initiated in compliance to the Ministry of Health regulations. Inputs from external experts were incorporated in fifty cases (72 percent). However, inputs from other stakeholders like patient organizations were not mentioned in any of the documents reviewed. The disclosure of sources of data was found in fifty-eight cases (84 percent). Almost all appraisals, sixty-eight (98 percent), had non-technical summaries for public end-users. No appraisal had a short summary in English and only nine reports (Reference Kristensen, Garrido, Kristensen, Nielsen and Busse13 percent) discussed the implications of the results and proposals for future action. Only four reports (6 percent) clearly stated their conclusions. The average score for the transparency criteria was 63 percent.
Quality. In sixty-seven cases (97 percent), the target disease was described adequately with background information on the disease, symptomatology, epidemiology, and prognosis. However, it is in only sixteen reports (23 percent) that evaluation and justification of the choice of the target population was included. With regard to defining comparators used in the appraisal, forty-seven reports (68 percent) gave information on the standard treatment, while only eleven (16 percent) elaborated on the appropriateness of the comparator/s used. Furthermore, almost all (97 percent) provided technical descriptions of the drug including conditions for use.
All appraisals evaluated the efficacy data. Additional remarks were noted in thirty-six cases (52 percent). Six reports received a positive comment. In the remaining thirty-two reports, the most frequent issue raised was a poor quality of data (Figure 1). HTA reports with a negative recommendation were mainly scrutinized at the Consultative Council's meetings with regard to efficacy.
Safety aspects were covered by sixty-eight reports (99 percent). Seventeen cases were further scrutinized where the most common issue found was selection of data, that is, insufficient evidence (Figure 2). Other issues raised included the lack of data from observational studies and poor data quality. Most remarks accompanied HTA reports with a negative recommendation.
Cost-effectiveness analyses results were included in sixty-seven cases (97 percent). Of these, thirty-eight reports had issues concerning methodology of the performed analysis, more precisely its quality. Critical comments were made with respect to HTA reports given a negative recommendation (Figure 3).
Figure 3. The Consultative Council's comments regarding cost-effectiveness analysis of appraised drug technologies in 2008.
Budget impact analyses were reported in sixty-six cases (96 percent). Of these, thirty-three were further deliberated because of some issues. The concerns about assumptions (model quality) and incorrect estimation of the target population were discussed at most occasions. Again, the comments were made alongside a negative recommendation (Figure 4).
Figure 4. The Consultative Council's comments regarding budget impact analysis of appraised drug technologies in 2008.
Social aspects were considered in four reports (6 percent) and organizational aspects in two reports (3 percent). A discussion on the long- and short-term consequences of the new technology (organizational consequences) is missing in all submissions.
In summary, the average score for eight quality criteria was 66 percent.
It should be underlined that a submission with a positive recommendation had an average of 1.7 issues raised during the deliberations while submissions receiving a negative recommendation had an average of 3.2 issues.
DISCUSSION
The present study reviewed all published HTA appraisal documents, reports, and deliberations on pharmaceutical products for 2008 on the AHTAPol Web site using standards of transparency and quality as defined by the INAHTA and ECHTA. The review of sixty-nine reports showed that criteria for quality were more consistently met than criteria for transparency. For transparency, most reports fulfilled five of the nine criteria (84 percent to 100 percent) but were wanting in providing English summaries, clarity of conclusions, and further recommendations to be taken. Also notable is the inadequacy of inputs from other relevant stakeholders. For quality, 97 percent to 100 percent of the reports included six of the eight criteria, while only 68 percent gave adequate information on the comparator/s being used in the appraisal. Ethical and social consequences as well as the consequences of drug inclusion to the healthcare system are not systematically considered in the reports. Most of the queries raised revolved around quality of evidence for drug efficacy, adequacy of the target population, and the methodological quality of the economic analyses.
This study reviewed published appraisal reports only after a year of implementation of the Polish HTA guidelines. Even at this stage of development, results show that AHTAPol has performed satisfactorily. Most HTA reports fulfilled the international criteria for transparency and quality. However, improvements are needed in the completeness and comprehensiveness of the submitted information. The justification for the choice of target population and the use of the appropriate comparator/s need more emphasis. Other elements that need to be strengthened include the quality and sufficiency of the evidence being presented and the methodological rigor of the economic consequences of the drug. There were several appraisals that received positive recommendations despite questions on inadequacy of several critical elements. The bases for such decisions are not apparent from the published documents.
Assessment processes across HTA organizations may vary. Differences in the diseases being covered, the types of technologies being assessed, and the assessment methods used by the appraisal body and the recommendations made, account for variations (Reference Garcia-Altes, Ondategui-Parra and Neumann10). This variability is also observed in AHTAPol, thereby necessitating standardized and explicit processes. Furthermore, other aspects of health technology influence policy decisions. Social, political, and ethical considerations often lead to trade-offs between scientific evidence and real-world situations resulting to a need to be consistent and transparent in the whole appraisal process (Reference Lehoux and Blume15).
This study has its limitations. The number of cases reviewed might not be enough to give a thorough picture of AHTAPol's performance. Furthermore, only documents published online were reviewed. An assumption was made that items not covered in the published documents were not appraised. There is a possibility that some of these missing topics might have been discussed but were not recorded and published. The online documents may not also provide a comprehensive understanding of how standards of transparency and quality, as defined in this study, were considered during the appraisal process. Another limitation is the way the HTA score card was constructed. Each criterion was given equal weight. It has been reported on many occasions that weights given to the evidence being put forward and public interest play a role in reimbursement decisions. Finally, this study considered conditional and restricted recommendations as positive recommendation and did not analyze them separately. This oversimplification might lead to biased results. A published study using a pooled sample of HTA evaluations revealed that different kinds of issues were raised alongside negative or positive HTA recommendations (Reference Bending, Kruger, Hutton and McGrath5).
Despite its weaknesses, this study not only gives an insight into the performance of the AHTAPol in 2008 but generates valuable proposals for the improvement of the Polish appraisal process and for the delineation of roles of its stakeholders—the HTA users and the HTA doers.
Three recommendations are directed to HTA users in Poland. First, to improve the quality of the HTA process, HTA users should engage in careful and comprehensive study of the organizational and social consequences of new drug technologies. Expected health outcomes can only be achieved if the healthcare system is prepared for the introduction of such technology (organizational consequences). Instances when healthcare professionals must receive adequate training for the adoption of the new technology or when additional investments would be beneficial in securing the safe and efficient utilization of the new technology may occur. Furthermore, social preferences must be incorporated in the decision-making process (social/ethical consequences). Subjective attitudes toward a given health technology such as low insight into the disease or low satisfaction with treatment affect compliance thereby creating a potential barrier to the implementation of the new technology (Reference Velasco-Garrido and Busse23). This omission is highlighted by the study.
Second, to enhance the transparency of the HTA process, there is a need to improve the way HTA recommendations are given. In many instances, deliberations, conclusions, and suggestions for future action are missing from the reports. A standard layout may be introduced to ensure the fulfillment of specific transparency standards like the template used by the Scottish Medicines Consortium, for example (www.scottishmedicines.org.uk).
Finally, a better integration of all the main players must be aggressively pursued to improve policy making, practical relevance of the process, and to ensure more accountability for the decisions taken. The results of this study indicate clearly that there is limited input from external stakeholders in the HTA process in Poland. An improvement in the engagement of the public assures its quality, validity, and accuracy (Reference Kristensen, Garrido, Kristensen, Nielsen and Busse13). In addition, the involvement of the manufacturers makes the appraisal process more effective and quicker. Any outstanding queries regarding available evidence can be addressed during an evaluation of a submission.
For HTA doers in Poland, three recommendations are forwarded. First, HTA doers must give careful attention to the submitted efficacy and safety evidence. This study found that efficacy is indeed the most frequently appraised subject from the list of key evidence, similar to what has been previously reported (Reference Draborg7). In the study, poor quality of data was the most frequent comment. It is important that both the methods for data collection and synthesis are appropriately presented. The comprehensiveness and completeness of the efficacy and safety data must be considered to avoid concerns over data quality. Safety issues must also be presented in a comprehensive and informative way. As noted in this study, adverse events reported in the randomized controlled trials including those from observational studies must be provided.
Second, a thorough study of the economic consequences of a technology is needed. The adoption of a new technology can be a major driver leading to an increase in healthcare costs (Reference Becker4). In the reviewed HTA reports, concerns over the quality of the cost-effectiveness analyses were raised. An economic evaluation is a means of supporting a system objective of maximizing population health gains from the available budget (Reference Sculpher21). Although the AHTAPoL guidelines allow for the application of different types of economic approaches, the assumptions made must be justified and potential limitations discussed thoroughly (Reference Busse, Orvain, Velasco and Perleth6;Reference Evers, Goossens, de Vet, van Tulder and Ament9;Reference Hailey11). In this study, there were cases where the time horizon was regarded as inappropriately defined, where not all relevant costs were incorporated or where the calculation of uncertainty was questioned. The time horizon chosen for any economic model should be consistent with the duration of time that treatment effects may be observed (Reference Schulpher, Fenwick and Claxton20). To be credible, all relevant costs must be considered and the outcome variables properly determined. Sensitivity analysis would be an important tool for judging the robustness and the implications of results (Reference Evers, Goossens, de Vet, van Tulder and Ament9;Reference Hilman12).
Finally, the correct estimation of the target population and market shares of particular drug technologies must be fully appreciated and accurately preformed. These are critical in budget impact analyses as errors may lead to over/underestimation.
The International Working Group for HTA Advancement recently investigated the extent to which the key principles for improved health technology assessment were supported and used by fourteen HTA organizations worldwide (Reference Neumann, Drummond and Jönsson18). They found that the key HTA principles were supported and used to various degrees by these HTA organizations. They also identified aspects for improvement which included good HTA practices in terms of transparency and being unbiased plus the need for monitoring of the implementation of HTA findings. AHTAPol's performance in this aspect would be interesting as future study.
In conclusion, the study demonstrated that AHTAPol has been fairly successful in the implementation of the HTA process in Poland and in its integration into the decision-making process for drug reimbursement. Using internationally developed standards of transparency and quality as measures of performance, the Polish HTA reports have fulfilled these to some extent. Recommendations are forwarded for both HTA users and doers for the improvement of the HTA process. After all, active engagement and commitment from both HTA users and doers in the appraisal process are critical to the achievement of optimal allocation of scarce resources in the Polish healthcare system.
CONTACT INFORMATION
Katarzyna Kolasa, PhD (kkolasa@wum.edu.pl), Senior Researcher, Department of Pharmacoeconomics, Warsaw Medical University, Pawinskiego 3a, 02-106 Waszaw, Poland
Malgorzata Dziomdziora, MA, MIM (mdziomdziora@gmail.com), Pricing & Reimbursement Specialist, Employers' Union of Innovative Pharmaceutical Companies in Poland, 17 Pulawska Street, 02-515 Warsaw, Poland
Liberty Fajutrao, MD, MSc (lfajutrao@gmail.com), Consultant, Foundation for the Advancement of Clinical Epidemiology (FACE), Dr. Paz Mendoza Bldg, Pedro Gil St., Ermiro, Manila, 1000 Philippines
CONFLICT OF INTEREST
K Kolasa reports having no potential conflicts of interest, M Dziomdziora is employed by the Employers' Union of Innovative Pharmaceutical Companies INFARMA and L Fajutrao is employed by AstraZeneca and has stock options in the company.
