Are value judgments important in health technology assessment (HTA)? A recent study indicates that this is the case: 90 percent of 104 responding HTA experts, publishing in this journal between 2005 and 2007, agreed that healthcare decisions involve value judgments and that ethical analysis is important to HTA (Reference Arellano, Willett and Borry1). However, the response rate was 16 percent, and respondents’ views on the role and place of value judgments were not clear. Traditionally, HTA and evidence-based medicine (EBM) have been considered to be a scientific and value-free endeavor (Reference Wallay2). Ethicists working in the field of HTA experience that a significant number of HTA experts think that value judgments threaten the presumably “objective scientific” basis of HTA, making it subjective, relative, and unreliable, as a distorting “bias.” Value judgments, and in particular social value judgments, are frequently considered to be external to HTA, and are best addressed by experts (in ethics) and decision makers after the “real” HTA is finished. This “externalist” conception of value judgments in HTA is illustrated in Figure 1.
Figure 1. Adapted from Tunis SR. Reflections on science, judgment, and value in evidence-based decision making: A conversation with David Eddy. Health Aff. 2007;26(Reference Hennen4) w500-w515, with permission
Although the clear distinction between facts and values is difficult to defend philosophically (Reference Putnam3), it appears to be strongly present in the current concept and use of HTA, for example, in the distinction between assessment and appraisal, between systematic review and personal preferences, between scientific judgments (e.g., the EBM analysis) and the social judgments. While parliamentary technology assessment (PTA) explicitly addresses value issues, HTA has been more reluctant to do so (Reference Hennen4). One may argue that there is within the HTA community a “Cartesian Angst” a “fear of ending in relativism and nihilism when one departs from the analytical-rational scientific tradition that has dominated Western science since Descartes” (Reference Flyvbjerg5) that perpetuates this reluctance. From the externalist perspective, value judgments represent a “bias” to be avoided or eliminated.
In this study, we reject the hypothesis that HTA can be free of value judgments. We will do so by illustrating how value judgments are necessarily involved in various crucial steps of the HTA process: identification, priority setting, assessment, appraisal, and implementation in policy and practice. Furthermore, we will argue that value judgments are of constitutive importance to HTA. Evaluation of technology involves value judgments as does the involvement of different stakeholders in the HTA process, such as scientists, decision makers, patient representatives, health professionals, and industry representatives. Moreover, making explicit the value judgments implicit in HTA (e.g., in the framing of the research question) as well as in the appraisal phase (weighing the information provided) can make HTA more open, transparent, and trustworthy, as well as promoting accountable and robust decision making.
METHODS
A panel of international HTA experts (twelve from nine countries) specialized in applying methodology in ethics for HTA met during a 2-day workshop on methodology in ethics for HTA in Cologne on January 19–20, 2012. The workshop was hosted by The Institute for Quality and Efficiency in Healthcare (IQWiG) and organized by the INAHTA/HTAi Ethics Interest Sub-Group on Ethical Issues in HTA (EISG). All participants contributed to the workshop, presenting their experiences with how value issues were acknowledged and addressed in HTA. The role of value judgments and approaches to explicitly address value judgments were discussed. In particular, the panel of experts scrutinized the various parts of the HTA process for implicit value judgments. Identified value judgments were then analyzed critically with respect to their role for the accountability of HTA results. The final results were elaborated after four rounds of deliberation through e-mail correspondence.
RESULTS
Value Judgments in Conducting HTA
Value judgments are judgments about what is good. However, things can be good in many ways. They can be good in and by themselves (intrinsic values) or because they result in something good (extrinsic values). The objective of health technology (HT), obtaining health, is considered by many to be an intrinsic value, while HT can be of extrinsic value in so far as it results in something good. There are many types of goodness, such as scientific, economic, technical, medical, professional, aesthetic, and moral goodness (Reference Meggle and Nida-Rümelin6).
As HTA is an evaluation of the goodness of an HT, it must consist of value judgments. In HTA we strive to use scientifically good (i.e., robust) methods, implement medically good outcomes (e.g., better health), in morally good ways, to facilitate socially acceptable decisions, at bearable costs. In this we make moral, methodological, legal, economic, and social value judgments, and sometimes they are intertwined to such an extent that it is difficult to tell them apart. Values are also at the core of EBM (Reference Molewijk, Stiggelbout, Otten, Dupuis and Kievit7). Setting limits to p-values and confidence intervals are based on (epistemological) value judgments. This means that values and value judgments are constitutive to HTA, both in assessment and in appraisal (Reference Hofmann8;Reference Hofmann9). Moreover, value judgments are at play already before assessment, that is, in the decision to conduct a HTA, in the selection of technologies to be assessed, as well as in the formulation of research questions.
The following values have been identified in the HTA process (Reference Hofmann9).
Moral values. For example, to help persons who require assistance in situations that are undesirable to them and that can be related to conditions considered to be bodily or mentally harmful.
Value of assessing technology. It is believed that HTA is a good way to assess healthcare technology, and we need no proof thereof. That is, HTA has a positive value.
Selection of technology to assess. As already indicated, the prioritization of which technology to assess is as evaluative as is the prioritization to which the assessment leads.
Values of knowledge. There are epistemological values in HTA, for example, goodness of: (i) inquiry: reproducibility, exactitude, completeness, consistency and coherence, as well as; (ii) study design hierarchy (e.g., randomized clinical trials (RCTs) over observational studies); (iii) study quality (e.g., according to Grading of Recommendations Assessment, Development and Evaluation, GRADE); (iv) endpoints and outcome measures (definition, selection, and construction of endpoints); (v)
Comparator. (selection).
Social values. Values related to the implications of implementing the HT for the patients’ good life and for the communal welfare.
Technological values: Technology is related to values on several levels. See Table 1.
Table 1. Values Related to Technology, Using Diagnostic Ultrasound as an Example
Note. From Hofmann B. When means become ends: Technology producing values. seminar.net 2006; 2(Reference Wallay2): 1–12.
Value Judgments in Selection and Presentation of HTA Results
As the selection of studies to include in a systematic review is based on value judgments, so is the presentation of results and the summary or synthesis of information. How the results are framed, is based on value judgments: which end-points we choose; how do we measure the end-points; which results (studies) we include; how we balance burdens and benefits, and what we include (and exclude) in the summary.
Interpreting qualitative measures, as given in legislation or guidelines also involves value judgments, for example, requirements of “additional benefit” in order for a HT to be implemented, may be represented by thresholds or intervals by HTA agencies. Moreover, such thresholds may vary between outcome measures, for example, between “hard” and “soft” end-points. Such interpretations and weightings are based on value judgments, which are not always made explicit.
Value Judgments in Economic Analysis of HTA
The measurement of societal preferences for the distribution of health gains has always played an important role in health economics. However, the implicitly assumed desire (preferences) to maximize health in economic evaluations may not correspond to how the majority in a society think health ought to be distributed (values), a fact that bears directly on the objective of resource allocation (Reference Nord10). The question is whether it is really health the society wants to maximize or whether it is something else, for example, capabilities. If so, health economic evaluations, that most often still use health as a denominator in the cost-effectiveness ratio, are not the right decision-making tool.
Cost-effectiveness analyses (CEA) presupposes that “a QALY is a QALY,” equally weighted across individuals which is a quasi-egalitarian value judgment. This value judgment is questionable in many circumstances and correcting for this deficiency is not straightforward.
Cookson et al. (Reference Cookson, Drummond and Weatherly11) presented four approaches to incorporating equity considerations into economic evaluations: (i) provide factual information about equity, (ii) perform an equity impact assessment, for example, using health inequality indices, (iii) analyze the opportunity cost of equity, and (iv) assign equity weights to health outcomes in the economic evaluations. The level of quantification of the equity concerns differs, with the first approach having the lowest and the fourth approach having the highest level of quantification. It should be noted that any weighting system for QALYs in CEAs will be incomplete or complicate the analysis, as there are so many potential dimensions of equity. A fifth approach, is to use fully quantified multi-criteria decision analysis (MCDA) (Reference Johri and Norheim12).
The technical development of CEA also involves important value judgments through the methodological choices that are made at different stages during the analysis. For illustration, we highlight three such decision points: the choice of perspective, the target population, and the outcome measure.
The societal perspective is often explicitly argued for, although other perspectives are frequently used, often for legitimate reasons such as concerns about the ethical implications of methodological limitations for measuring costs and outcomes from a societal point of view (Reference Culyer13). However, by taking a healthcare payers’ perspective, it is judged that societal costs outside the healthcare sector, for example, productivity losses, should not determine a healthcare reimbursement decision. However, when productivity losses are included in the analysis, the choice of the unit cost of a day off of work suggests a value judgment. For example, if the wage of the person on sick leave is taken as the relevant unit cost, the productivity loss of a person performing higher paid work is valued more highly than the productivity loss of a person performing lower paid work.
For the definition of the target population, there is a distinction between clinical medicine and public health interventions. In clinical medicine, the societal importance of equality makes it unacceptable to discriminate between target groups by income, age, sex, or ethnicity in the provision of health care (Reference Cookson, Drummond and Weatherly11). Public health interventions, in contrast, may be oriented toward reducing health inequalities, so targeting of interventions is considered acceptable up to a certain point (Reference Cookson, Drummond and Weatherly11). Nevertheless, economic evaluations often distinguish between sub-groups for both types of interventions to demonstrate differential cost-effectiveness across population sub-groups. CEA thus show what the choice for equal access suggests in terms of reduced cost-effectiveness, but typically fail to identify, measure, and value health inequality impacts. As a result, decisions based on such information (or lack of it) rely on implicit value judgments about how equity ought to be conceptualized and balanced with efficiency.
Another value based choice in the economic evaluation process concerns the outcome measure used in the cost-effectiveness ratio. The two most frequently used generic outcome measures are life years gained (LYG) and QALYs gained. Although the impact of health interventions on health-related quality of life (HRQoL) is generally considered important, some may prefer LYG as an outcome measure for CEA because of its objective nature and the lack of consensus on methods for measuring HRQoL. Because the valuation of health states differs between those who experience a health state directly and those who do not, the outcome of the cost-effectiveness analysis will differ depending on whose values are used. Two approaches to this problem exist: either an explicit choice is made regarding whose values should count or the impact on HRQoL needs to be considered separately, if it is considered valuable at all. The QALY approach also embodies value judgments regarding the trade-off between HRQoL (and its various dimensions) and length of life (Reference Schwappach14). For example, 2 years of extra life expectancy at a HRQoL of 0.5 (1 QALY) are valued equal to 10 years of extra life expectancy at a HRQoL of 0.1 (also 1 QALY).
Ethical, Legal, and Social Values
“Ethical, legal, and social issues” (ELSIs) are together with organizational issues crucial to the assessment of HTs. ELSIs address the conditions and consequences of the implementation and use of HTs, related to the individual, to specific groups, and to society at large. This means ELSIs are closely related to societal settings and societal dynamics, and as such involve a wide range of moral, legal, and other social value judgments. For the assessment of many HTs these value judgments are intertwined, but as will be described below, for certain parts of the HTA process or for particular HTs, it is fruitful to differentiate between moral, legal, and social value judgments.
Many new technologies, such as preimplantation genetic diagnosis, have posed basic moral challenges and sparked debate. Other technologies have challenged religious, social, or cultural conceptions, for example, cochlear implants have incited debates on whether deafness is a functional defect or a characteristic of people sharing a special (sign) language. Some technologies have challenged (and changed) the legal framework in some countries, such as genetic testing. Organizational issues have come to focus as a result of other technologies.
However, value judgments are not only present in controversial technologies, but also present, though more subtly, with traditional HTs. Hence, analyses of ELSIs may highlight inherent and hidden value judgments in noncontroversial HTs. How much, and which, ELSIs and organizational issues that are addressed, may vary between different countries. It may be important to be aware of such contextual differences to promote sound reasoning and robust decision making.
Even though the importance of ELSIs may be generally accepted, only few institutions are systematically integrating ELSIs and organizational issues in their assessments (Reference Burls, Caron and de Langavant15–Reference Hofmann17). Moreover, ILSIs are addressed selectively, with implicit selection criteria. Such approaches risks failing to identify and address the implicit and covert values involved in implementing HTs. To avoid such shortcomings and to promote openness and transparency in the HTA process, ELSIs and organizational issues should be addressed in a systematic way.
Legal issues are of significance because (a) HTs are subject to rules and regulations (e.g., patents/licenses/approvals), (b) HT assessment is subject to legislation, and (c) patients’ basic rights to health care are given by law (18). Values and norms are inherent in legislation and are the drivers for changing existing laws and regulations. Legal value judgments may be contextual as laws may be regional or national.
Value Judgments in Appraisal and Decision Making
The appraisal process considers the outputs of the assessment within the context of additional information supplied by relevant parties for the decision-making process. Nowadays, some European countries have explicit processes in place making use of multi-stakeholder consultations (e.g., for coverage decision making in the Netherlands) or direct stakeholder involvement in appraisal committees (e.g., for pharmaceutical reimbursement in Belgium). Various methods for identifying and handling values in the appraisal stage exist, such as the UK National Health Service (NHS) guidance on social values and the Ontario Health Technology Advisory Committee (OHTAC)'s use of Daniels and Sabin's “accountability for reasonableness” framework. However, other less formalized or explicit decision-making processes are also used in making reimbursement decisions on HTs (Reference Velasco Garrido, Borlum Kristensen, Palmhoj Nielsen and Busse19). Hence, how to define appraisal and the assessment-appraisal border, as well as how to organize the appraisal process, involves a series of value judgments, such as who should do the appraisal, who should be involved and how should stakeholders be involved (Reference Green20).
In the appraisal process a judgment is made on the relative importance of a range of factors that differ from appraisal to appraisal. These factors could be, for example, burden of disease, extent of individual responsibility, effectiveness, cost-effectiveness, and budget impact. However, such factors are valued differently by patient representatives, health professionals, policy makers, payers, academic researchers, industry members, carers, citizens, etc. involved in appraisal committees. When stakeholders make value judgments with regard to the aforementioned factors, it may result in different effects on resource allocation. This has been demonstrated in a study of selected HTAs in The Netherlands, including the application of a growth factor (GM-CSF) as an adjunct to chemotherapy in treating elderly people with acute leukemia, evaluation of cervix smears by using a semi-automatic system (PAPNET), infertility diagnostics, lung transplantation, extra-corporal membrane oxygenation in neonates, and cochlear implants by prelingual deaf children (Reference Oortwijn and van der Wilt21).
Value judgments are constitutive in decision making: “.–.–. some person or some organization needs to make a judgment about whether the quality of a particular body of evidence is adequate to make a particular decision” (Reference Tunis22). However, the principles and values governing the aspects of resource allocation, rationing, and prioritization vary with the context. Fair allocation decisions are expected to concern all relevant stakeholders. However, the principles and considerations that people think are important or relevant differ. An analysis of selected debates about inclusion of HTs in the benefit package in The Netherlands showed that stakeholders interpret “necessary care” in at least three different ways (individual medical need, effective and cost-effective health care, and [normal] social participation); this has led to confusion and questions about fair resource allocation (Reference Hoedemaekers and Oortwijn23). Open and explicit procedures for decision making are requirements for legitimate decision-making processes. In addition, whether actors in decision making are equal partners and whether their arguments are assessed on a basis of convincing evidence and sound arguments, is related to values, for example, institutional and social values. In addition, processes are valued for being transparent, efficient, and sustainable. “Neither science nor economics will resolve the pain of choice. The best we can hope for is to strive to improve the process by which we reach the decisions” (Reference Klein, Day and Redmayne24).
DISCUSSION
A Value Judgment Is not a “Bias”
Our scrutiny of the various parts of an HTA process shows how value judgments permeate all levels of what is considered to be the scientific endeavor of HTA (assessment, appraisal, and HTA-based decision making). Our analysis also reveals that the value judgments are not alien to the HTA process. On the contrary they are constitutive to HTA, for example, selecting end-points and deciding on comparators are natural parts of the HTA process.
Many of the value judgments are implicit or tacit, and, by not making them explicit, the illusion of scientific objectivity and neutrality is reinforced. However, by leaving these judgments implicit, they may cloak important value issues and controversies and, as such, frame or “bias” the decision-making process. Hence, ignoring value judgments in HTA may make it more “biased” than addressing them. Accordingly, we contend that making value judgments explicit can improve the quality of the HTA and the decision-making process.
For example, in the case of treatments for adult obesity, the ethics analysis revealed implicit assumptions and value judgments involved in framing and conducting the assessment. A consequence was that certainties in the policy analysis became uncertain. This can be frustrating as we yearn for clear facts for clear-cut decisions. However, the reasoning of the policy analysis may be made more accountable and valid because it then more accurately reflects the actual situation. The reasoning may be sounder because implicit assumptions were made apparent and could then be the focus of discussion. Stakeholders can then have an equal opportunity to understand what is at stake and to participate in the discussions and reflect on the impact of a technology. As a result of making value judgments explicit, assumptions are more likely to be justified and the assumptions and the claims they buttress are more likely to be connected appropriately.
Therefore, making the value judgments of the HTA process and the “scientific judgments” of HTA explicit, clear, and transparent can provide the basis for more sound and sustainable decisions and, as such, enrich HTA.
How to Address Value Judgments?
As we have tried to show, value judgments are at the core of HTA, they are part of its raison d’être and its entire process. To highlight the values of the facts and processes of HTA is important. However, to question the strict border of science and ethics, assessment and appraisal, does not mean that separating assessment, appraisal and decision making does not make sense. They represent different kinds of value judgment in different contexts.
On the basic assessment level, it seems useful to make the process of defining, constructing, and evaluating effects and end-points explicit and discuss the reasons why specific comparators and thresholds were chosen and how. To make well-considered value judgments explicit on different levels, calls for specific approaches. Methods developed for doing ethics in HTA (Reference Cookson, Drummond and Weatherly11;Reference Droste and Rixen16) may be useful for highlighting moral (and legal) value judgments in the process of conducting HTA. Moreover, several methods are available for addressing values and arguments, see Table 2. The value base of HT may be as important as its evidence base and core values may be elaborated for HTA, such as universal access, freedom of choice, and quality care (Reference Bombard, Abelson, Simeonov and Gauvin25). Macro-level reflections on the general values of health care (health, wellbeing, welfare) and its philosophical underpinnings, such as equity, equality (and its specific forms), maximization of health or preferences or social values are needed and should be more widely known and explicitly used. This can be promoted by guidelines on how and why to conduct HTA and in appraisals and decision making.
Table 2. Guidance Literature on How to Identify and Address Values in HTA
In economic evaluations, no quick fixes exist. There seems to be no consensus on how to deal with the key elements of equity. Therefore, scenarios could be presented in the assessment to show the results of a cost-effectiveness analysis based on choices reflecting different value judgments. The key issue will be explicitness about the value assumptions and judgments made during an assessment process, in other words, to declare the ethical framework underlying the economic analysis.
A clear and transparent exposition of value judgments made during the HTA could be matched with the relevant societal values. As such, the appraisal process can remain deliberative, while the assessment can remain descriptive. It might be useful to first acquire a full understanding of the decision-making context and societal values and to figure out its consequences for the perspective, outcome measures, the discount rates and equity issues to be considered (Reference Culyer13). Such an approach would reduce the number of scenarios to consider in a particular context.
Less sanguinely, we acknowledge that to include all of these reflections into an HTA may render it rather extensive and less user-friendly, as well as less transferable to other countries and contexts. However, not all value judgments may be as pressing and challenging in every assessment. Moreover, for HTs involving a wide range of value judgments, the awareness of this complexity may be painful to decision makers. However, the complexity of value judgments may better reflect what is at stake. A “quick fix” is most often neither quick nor a real fix.
Moreover, another group of experts could identify and emphasize other value judgments, and our panel would be able to continue the work and identify additional value judgments. They (or we) may also suggest other methods for addressing the value judgments. However, all value judgments identified in this article are found in the HTA literature. Hence, they should not be unfamiliar to a well-informed HTA expert. Nevertheless, we acknowledge that more research on how to highlight and integrate value judgments in HTA is needed.
CONCLUSIONS: VALUE JUDGMENT IN HTA: FROM FOE TO FRIEND
We started with a notion of Cartesian Angst (anxiety) for value judgments which we have experienced with several of our HTA colleagues and which may help to explain the difficulty with addressing value judgments in HTA and with integrating ELSIs in HTA. Our analysis identifies value judgments at all levels of HTA, and reveals how they are constitutive to HTA. We suggest a relief of the Cartesian Angst: by accepting value judgments, making them explicit, and considering them openly, the HTA process may become more accountable. We suggest a move from an “externalist” conception of value judgments that views value judgments as something “added” to the results of an HTA to an “internalist” one that acknowledges and openly addresses value judgments as they arise within the entire HTA process. We also refer to several ways to address the inherent value judgments in practice. Accordingly, acknowledging and addressing value judgments may improve HTA.
CONTACT INFORMATION
Bjørn Hofmann, PhD (bjoern.hofmann@hig.no), Professor, University College of Gjøvik, PO Box 1, N-2802 Gjøvik, Norway
Irina Cleemput, Professor, Hasselt University and Belgian, Health Care Knowledge Centre (KCE), Brussels, Belgium
Kenneth Bond, MA, Director, Strategic Initiatives, Canadian Agency for Drugs and Technologies in Health, 600-865 Carling Avenue, Ottawa, Ontario K1S 5S8, Canada
Tanja Krones, MD, PhD, University Hospital/University of Zürich, Institute of Biomedical Ethics and History, Switzerland
Sigrid Droste, Dr. rer. biol. hum., Department of Quality in Healthcare, Institute for Quality and Efficiency in Healthcare (IQWiG), Mediapark 8, 50670 Cologne, Germany
Dario Sacchini, Assistant Professor, Institute of Bioethics, “Agostino Gemelli” School of Medicine, Università Cattolica del Sacro Cuore, Rome, Italy
Wija Oortwijn, PhD, Ecorys Netherlands, Rotterdam, the Netherlands
CONFLICTS OF INTEREST
The authors report they have no potential conflicts of interest.
