Health technology assessment (HTA) should facilitate systematic and transparent assessment and appraisal of decision making for coverage of health technologies (Reference Velasco, Gerhardus, Røttingen and Busse28). A good decision process involves not only the best available scientific evidence but also a prudent choice of evaluation criteria and a deliberative decision-making procedure that is participative and transparent. Processes should focus on technologies that are likely to provide high value. Not all decision-making processes use formal HTAs. It is fair to ask whether the involvement of HTA in health policy making contributes to fair and legitimate decision processes and structured and transparent decision making. Do the steps of decision processes differ depending on whether an HTA exists when a coverage decision is made?
Research has previously tried to assess the role and impact of HTA and related programs on health policy processes (Reference Gerhardus, Dorendorf, Rottingen, Santamera, Velasco, Kristensen, Nielsen and Busse8;Reference Hanney, Buxton, Green, Coulson and Raftery11). Other health policy research studies have assessed the criteria used for appraisal, the decision outcome and assessment methods (Reference Vuorenkoski, Toiviainen and Hemminki29). None of these studies explicitly examined the interdependence between HTA and coverage decision across countries.
The aim of this article was to explore the association between HTA and decision-making processes for newborn screening (NBS) in European countries and generate a set of hypotheses. NBS includes several promising technologies that are relevant to third party payers. In particular, tandem mass spectrometry (MS/MS) allows screening of multiple metabolic disorders in one step. The results of this exploratory study may shed light on how HTA contributes to coverage decisions.
METHODS
To generate hypotheses on the association between HTA and coverage decision making, we performed a literature search in the following databases: MEDLINE, EMBASE, and Web of Science. The purpose was to identify all studies that have explicitly addressed the role of HTA in coverage decision making, also by analyzing past coverage decisions. We searched titles and abstracts published after 1987 for references using a combination of the terms “health technology assessment”, “coverage”, “decision”, and “influence” including synonyms which was last updated April 7, 2011. No systematic abstraction of data and appraisal of articles was performed as this step aimed to retrieve qualitative information to generate statements on the role of HTA.
For structuring of the steps of decision processes, several conceptual frameworks have been stated (Reference Daniels and Sabin5;Reference Hutton, McGrath and Frybourg14;Reference Rogowski, Hartz and John23). Rogowski et al. describe the stylized elements of a decision process from the point where a technology enters a healthcare market to its diffusion into routine use (Reference Rogowski, Hartz and John23). This structure captures both process and appraisal criteria. The following steps appeared relevant: trigger of a specific decision process; stakeholder participation; publication of decision and related documents during or after the decision process; assessment by scientific methods like systematic literature review; and appraisal of whether or not the technology should be funded in the light of the available evidence along with further ethical or political criteria. A structured scheme including indicators and ordinal rankings for each step of the framework has been validated with a small number of decisions and expert discussions (Reference Fischer, Leidl and Rogowski7).
From studies identified in the literature review, key principles of HTA (Reference Neumann and Drummond27), and the steps of the framework by Rogowski et al. (Reference Rogowski, Hartz and John23), a group of statements was formulated that describe the association between HTA and steps of NBS decision making. As the publication of HTA and decision making mutually influence each other, no statement on the causality of associations was made. Table 1 provides an overview of the statements that were made for the case where an HTA was published.
Table 1. Steps of Decision Processes, Statements and Selected Operationalization
a All statements are made for the case where an HTA has been published.
A survey of recent NBS decisions in the European Union was conducted between August and December 2009. Experts were recruited through a scientific society (International Society of Neonatal Screening) and were either affiliated to NBS providers or third party payers of which forty-three completed the questionnaire (response rate 70 percent). Respondents were asked about decisions made since 2005 on screening for medium-chain acyl-CoA dehydrogenase deficiency (MCAD), cystic fibrosis, or congenital adrenal hyperplasia. If none of the conditions mentioned were identified, the survey also allowed for the analysis of other case studies. In this context, a case study was defined as screening of an inherited disorder (e.g., MCAD) using a specific test strategy (e.g., MS/MS). The survey was based on a web-based questionnaire where one question was stated for each indicator of the structured scheme which can be obtained from the authors upon request. A total of fifty-five decisions were reported from twenty-one countries.
Results of decisions that have been evaluated by two respondents showed that the inter-rater reliability is low and answers need to be validated to obtain accurate results. To ensure validity of statistical tests, the data set was restricted to those twenty-two decisions that had been cross-validated by two experts from each country through application of a Delphi procedure. Conflicts remained in approximately 5 percent of answers. Then, statements from the expert that had the closest relation to the decision were considered, for example, statements from third party payers were preferred to service providers. Table 2 shows the countries and decisions included in the present analysis.
Table 2. Analyzed Decision Options
a Expansion of number of newborns screened.
b Selective screening.
BIO, biotinidase deficiency; CAH, congenital adrenal hyperplasia; CF, cystic fibrosis; CH, congenital hypothyroidism; GALT, classical galactosemia (galactose-1-phosphate uridyltransferase deficiency); Hb SS, sickle cell disorders; MCAD, medium-chain acyl-CoA dehydrogenase deficiency; MSUD, maple syrup urine disease; PKU, phenylketonuria.
The indicator of HTA activity is the publication of an HTA report. The data set was split in two groups—decisions reporting the publication of an HTA or not. Related activity might have played a role but no other objective indicator could be drawn. Frequencies of categories for selected indicators describing the steps of decision processes were calculated. Due to the small sample size, a nonparametric method (two-sided Fisher's exact test) was applied to test for differences in distributions of frequencies of categorical variables between both groups. A two-sample Kolmogorov-Smirnov test was applied for comparison of the number of documents published and number of participating stakeholders, as no assumption about the distribution of variables could be made. This test assesses whether the distributions of variables between two samples differ statistically. For the step appraisal, answers from both respondents were considered. The sum of absolute deviations in the rating of relevance between both groups was additionally derived. To account for dependent observations, the test statistic of Fisher's exact test was corrected for the existence of two respondents. As the study objective was exploratory, no adjustment for multiple testing was made. Data evaluation was performed with SAS Version 9.2 (SAS Institute, Cary, NC).
RESULTS
The literature review retrieved fifty references excluding conference proceedings of which seventeen articles were identified that discussed the role of HTA in relation to coverage decision making. Two studies analyzed past decisions for single decision makers, but not in relation to the process of decision making (Reference Anis, Rahman and Schechter1;Reference Barbieri, Hawkins and Sculpher2). Other research discussed the role of HTA by conducting structured country comparisons, for example (Reference Levy, Mitton, Johnston, Harrigan and Briggs17;Reference Menon and Stafinski18;Reference Oortwijn, Mathijssen and Banta20;Reference Schwarzer and Siebert24), or provided case studies for single countries, for example (Reference Carlsson4;Reference Jonsson15), displayed general categories on the role of HTA in the innovation process (Reference Bührlen3) or considered HTA as one of many factors to accelerate technology diffusion due to increased publicity (Reference Packer, Simpson and Stevens21). Other authors elicited decision-maker's preferences (Reference Koopmanschap, Stolk and Koolman16) or examined the role of early HTA to inform policy decisions (Reference Hartz and John12). In addition to the published studies, an OECD survey was identified that gathered data on the role of HTA in decision processes for five technologies in twelve countries (19).
Among the twenty-two decisions on NBS technologies with two independent reports, an HTA report was available during or after the decision process for seven decisions (Reference Davies, Cronin and Gill6;13;Reference Pandor, Eastham, Beverley, Chilcott and Paisley22;Reference Seymour, Thomason and Chalmers25;26;Reference Zeuner, Ades and Karnon30). Table 3 displays an overview of the frequencies of the selected indicators grouped by existence of an HTA report. Table 4 shows the results of the Kolmogorov-Smirnov test.
Table 3. Overview of Results by Step of Coverage Decision Process and Existence of HTA Report for Decisions on NBS Technologies in Europe
a More than one answer was possible.
b See Table 2 for a detailed overview of decisions.
Table 4. Number of Participating Stakeholders and Published Documents
a See Table 2 for a detailed overview of decisions.
Trigger
The specification of explicit criteria by stakeholders was found to be associated with the publication of an HTA. All processes with HTA reports were initiated after the specification of screening criteria. In contrast, 40 percent of processes which had no HTA publication were started ad-hoc, without explicit criteria. In Slovenia, for example, the screening decision was said to be the result of the availability of appropriate screening equipment. Four countries reported explicit screening criteria. In the Netherlands, the Health Council developed its own criteria after reviewing the Wilson and Jungner criteria and other lists (13). In England, France, and Denmark, standing committees formally started the decision.
Participation
The mean number of involved stakeholders was larger where an HTA was conducted (Reference Carlsson4.Reference Grosse, Olney and Baily9 versus 3.4 in the non-HTA group). Compared with the group with HTA, only the industry and third party payers were involved more frequently where no HTA was existent. Significant differences in the frequency of participation could only be identified for patients and HTA agencies toward a higher participation in the group with HTA report.
Differences in the opposite direction were found with regard to the degree of involvement of stakeholders. Where an HTA was provided, stakeholders were involved in information provision (71 percent). On the contrary, almost 90 percent of stakeholders in the group without HTA were involved in voting on the decision option and only 13 percent provided information. Decision processes with an HTA were thus more participative in terms of inclusion of stakeholders but simultaneously less participative in terms of the degree of participation.
Publication
For all decisions, some kind of information was reported, the only exception being in Slovenia. The number of published documents per decision significantly differs between the group with and without HTA. In the group of decisions with HTA the frequency of published documentation was higher for all types of documentation. In the Netherlands, stakeholder comments and the assessment question from scoping was accessible, whereas in England, minutes of the appraisal meeting were available. For decisions without HTA report, most frequently only the decision outcome was reported. Overall, no information was made publicly available during the decision process in 64 percent of decisions. In the group with HTA 29 percent of decisions reported that no information had been made available compared with 80 percent in the group without HTA involvement.
Assessment
The effectiveness of NBS was assessed in all decisions. In 13 percent of processes without HTA report the assessment was solely based on expert opinion. For one decision, no information on the type of assessment of effectiveness was provided. In decisions with HTA report, the assessment of effectiveness was based on different scientific methods. Expert opinion and a review of the literature were always considered. For decisions including an HTA, the conduct of a systematic review of the literature could be validated. For the other group, it was not possible to determine whether the literature was reviewed systematically. In Denmark, it was stated that a quantitative meta-analysis had been conducted by the American College of Medical Genetics (ACMG), with results that were adjusted to the Danish situation (26). However, the ACMG report was based on expert opinion and did not meet the criteria for a systematic review (Reference Grosse, Rogowski and Ross10).
Costs or cost-effectiveness were reported to have been assessed in all decisions except for the decision in Slovenia. In one third of the decisions without HTA, the assessment was supposed to have included a cost-effectiveness analysis, but this was not documented. That term is often used loosely, and it was not possible to validate whether cost-effectiveness analyses were conducted according to best practice guidelines.
Appraisal
In Table 3, the aspects that appeared relevant or strongly relevant for the final decision outcome are displayed. For both groups the health gain from screening was considered to be most important criterion. It was considered strongly relevant in all decisions with HTA, compared with 60 percent of decisions without HTA. By comparing the sum of absolute deviations in the rating of relevance between both groups, the availability of treatment for the screened disease, budget impact, and health gain from testing were those aspects which were relatively more relevant in the group of decisions where an HTA was published. Although lobbying activities were reported to play a minor role in both groups, respective activities by the industry were relatively more relevant in the group without HTA. On the contrary, involvement of patients was more important in decisions with HTA report. Only a few aspects significantly differed between the groups: expected costs and lobbying activities by service providers. For many appraisal aspects, test statistics could not be gathered as there were zero observations in at least one cell of the two-by-two table. For an overview of test statistics, see Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc2011023. If results were adjusted for multiple testing, the significance level would drop below 0.0014. In this case, no difference between groups would be significant.
DISCUSSION
Some process steps for NBS decisions appear to have differed by whether an HTA report was prepared as part of the process. The existence of explicit criteria for selection of the technology in the trigger step was positively associated with the subsequent availability of an HTA report. Likewise, for an explicit rationale for the assessment question from the scoping process was significantly positively associated with the publication step. An implication is that if an HTA is conducted, the exact assessment question needs to be stated before the assessment.
Where an HTA was produced, service providers, government, HTA agencies, patients and academia were involved more often. To a great extent, this is likely to reflect stakeholder involvement in information provision. Involvement in the appraisal process did not differ depending on the conduction of an HTA. It seemingly guided stakeholders toward information provision during assessment instead of reducing the opportunity for single stakeholders to influence appraisal toward their interest. This may suggest that there is an association between the publication of an HTA and decision making toward higher transparency and other possibilities for stakeholder involvement. However, increased involvement has to be balanced against the apparently weaker mode of participation under an HTA regime (information provision versus voting).
In general, coverage decisions for newborn screening were based in part on an assessment of effectiveness, which should at least in part be based on a review of the literature. For decisions where an HTA was published, it could be validated that the literature review was systematic and based on a documented methodology. Although most decisions without HTA also reported the use of systematic reviews, it seems unlikely that systematic reviews were actually conducted in this group. All respondents reported that expert opinion was considered, which indicates that expert opinion was considered indispensible – independent from the existence of an HTA.
Contrary to prior expectation, no difference was found in the reported use of cost-effectiveness analysis relative to the presence of an HTA report. However, the quality of the assessments likely differed. More than half of the decision makers considered economic criteria of expected costs or cost-effectiveness as relevant criteria, regardless of the availability of HTA. Among them only one-eighth to one-fifth considered economic criteria to be strongly relevant.
If an HTA was conducted, decision makers appear to have considered tangible clinical criteria, for example, the health gain from screening, to be more strongly relevant, although this difference was not significant. The direction of distance is consistent with the expectation that HTA does not only suggest better quality decision processes in the sense of a greater degree of transparency and stakeholder participation (at least in terms of providing information), but also better criteria during assessment and appraisal in the sense of criteria which are typically targeted by scientific clinical studies.
Limitations
Although this is a larger sample of decision processes than has been considered previously, the results relied on a small data set. The number of decisions with HTA report was relatively higher in the selected than in the full data set (46 percent compared with 10 percent). Accordingly, the effect from HTA may be overestimated. The existence of an HTA report is a crude indicator for HTA activity. However, comparison of the results grouped by participation of an HTA agency showed no major difference. Furthermore, participation of an HTA agency in the decision process might overestimate effects as it does not suggest that the technology under consideration had been evaluated fully.
The issue of endogeneity should not be neglected. HTA informs decision makers and vice versa. Presumably, there are countries or regions with both an established participative and transparent decision processes and established HTA institutions while others lack both. This might potentially overestimate the influence from HTA activities. If a new HTA were set up in a country with a different tradition of health policy making, it might have little direct effect on policy processes.
This study assumed that decision making occurred at a single level for a given country. Decisions might be made at the national or regional government level and additionally, by healthcare providers and payers. Beyond this, coverage decisions always are subject to policy making so that decisions are influenced by healthcare system settings, available resources and stated priorities. For newborn screening, the prevalence of screened disorders further differs across populations. When describing single decision processes in detail, the particular context of decision making should be taken into account. However, this was not the subject here as we focused on the decision processes.
The findings on NBS decision processes are exploratory results that require confirmation, particularly for other technologies and in other countries. For exact estimation of the interdependencies, statistical methods other than nonparametric tests are needed. Instruments such as regression analysis were not applicable to address the association between HTA and decision process due to the scarcity of data and missing parameter distributions. More thorough and precise analyses will also require larger data sets which are time consuming to collect.
Implications
The results offer implications for policy makers on the role of HTA in coverage decisions. Undertaking an HTA during a decision process not only may disclose relevant evidence in an objective manner but more transparent and participative decision processes may be promoted simultaneously. Vice versa, participative and transparent decision processes might ensure the proper use of HTA methodology.
Furthermore, the use of HTA appeared to be associated with an increased role of clinical effectiveness in decision making. At least to some extent, HTA may also guide decision makers toward more reasonable appraisal criteria. The finding that HTA was not associated with greater use of appraisal criteria such as cost-effectiveness, as has been noted earlier (Reference Velasco, Gerhardus, Røttingen and Busse28), indicates that HTA is not a determining factor in their consideration. If policy makers wish to be able to make use of cost-effectiveness information for appraisal, they should explicitly demand such information. The major limitation, though, for the use of cost-effectiveness information in NBS decision making seems to be lack of demand for this information. Even if it is reported in HTA, it may not be used by decision makers, as has been documented (Reference Grosse, Olney and Baily9).
When an HTA was published, decision processes were always triggered by explicit criteria. This may indicate that the use of explicit criteria for triggering decisions may lead decision makers to commission an HTA because it can ensure that criteria relevant for triggering the process were appropriately assessed for the technology under investigation.
In addition to general implications, conclusions can be drawn for the use of HTA in NBS policy making. From eleven analyzed countries, only three had an HTA report provided in the context of the decision. In four countries without HTA reports, an HTA agency existed but did not produce a report because HTA was not a formal requirement in NBS decision processes or the decisions were made on regional levels (i.e., Flanders, Belgium). Decision makers may consider evidence from assessments produced in other countries as sufficient for their NBS decision making as was indicated by one respondent country, Denmark. Some of the qualitative results may be specific to NBS. Given that randomized clinical trials have not been conducted for NBS, with the exception of cystic fibrosis, the degree of scientific rigor is lower for NBS than for many technologies, such as pharmaceuticals.
Our findings of variability of perceptions among respondents of NBS policy process may have broader implications for research: equal processes can be perceived differently by different observers. Therefore, it is essential for future studies of the role of HTA to elicit input from multiple expert informants. Ideally, they should reflect different stakeholder groups to assure the validity of responses. This can shed light on the extent to a lack of agreement or transparency regarding policy-making processes.
CONCLUSION
This study examined the association between the publication of an HTA and the steps of decision processes in a structured and consistent manner for a distinct technological area. It explored associations between the steps of decision processes and publication of an HTA. The goal of HTA should be to ensure good decision processes for coverage of technologies. First evidence on possible associations between the HTA and process steps of decision making have been shown for selected steps and single indicators. Not only was HTA associated with the outcome of decision processes—as shown earlier—but results also suggest that decision processes themselves differ by the presence of an HTA in terms of stakeholder involvement and documentation of decisions.
Conducting an HTA may not necessarily be a guarantee of a deliberative decision process with higher transparency and improved possibilities of fair stakeholder participation, features that have been discussed earlier (19), but it appears to have positive associations with those attributes.
CONTACT INFORMATION
Katharina E. Fischer (Katharina.Fischer@helmholtz-muenchen.de), Helmholtz Zentrum München, German Research Center for Environmental Health, Institute of Health Economics and Health Care Management, Ingolstädter Landstr. 1, 85764 Neuherberg, Germany
Scott D. Grosse (sgg4@cdc.gov), U.S. Centers for Disease Control and Prevention, National Center on Birth Defects and Developmental Disabilities, Division of Blood Disorders, 1600 Clifton Road, Mail Stop E-64, Atlanta, Georgia 30333
Wolf H. Rogowski (Rogowski@helmholtz-muenchen.de), Helmholtz Zentrum München, German Research Center for Environmental Health, Institute of Health Economics and Health Care Management, Ingolstädter Landstr. 1, 85764 Neuherberg, Germany; Institute and Outpatient Clinic for Occupational, Social and Environmental Medicine, Clinical Center, Ludwig–Maximilians–University, Ziemssenstraße 1, 80336 Munich, Germany
CONFLICT OF INTEREST
Katharina Fischer and Wolf Rogowski report their institution has received a grant from the European Union's Seventh Framework program for this work. Scott Grosse has no potential conflicts of interest.
