While most people recognize that the opinion of the patient, his or her voice, should be central in decisions to authorize, reimburse, or prescribe a health intervention (Reference Facey, Boivin and Gracia1–Reference Frosch5), whether this is being meaningfully achieved may be open to question (Reference Agoritsas, Heen and Brandt6). This is most starkly illustrated by the stories of patients challenging regulators’ treatment access decisions to get them overturned. For instance, patients with multiple sclerosis successfully challenged the Food and Drug Administration (FDA) to get access to natalizumab (Reference Pollack7). Patient groups have used social media to promote FDA approval of treatments that have previously been voted against by external FDA advisors (Reference Fagerlin, Zikmund-Fisher and Ubel8;9).
Patients’ concerns continue to exist despite recent efforts by decision makers to incorporate patients into the reimbursement and regulatory decision-making process. The Patient-Centered Outcomes Research Institute (PCORI) is a U.S. institution established to fund research designed to improve patient care and outcomes through methods that bring the patient to the center of healthcare research and development (10). The European Medicines Agency (EMA) recently announced a pilot project to invite at least two patients to be involved in discussions on product specific benefit and risk evaluations with the Committee for Human Medicinal Products (CHMP). The CHMP is responsible for preparing opinions on issues and questions relating to evaluating medicines (11). Patient representatives have voting rights as part of FDA Advisory Committees that review new drug applications (12). Health technology assessment (HTA) agencies sometimes involve patient groups in committees that decide whether drugs should be reimbursed at the asking price, but their involvement is not considered systematic or widespread (Reference Facey, Boivin and Gracia1).
The incongruity between patients’ challenging authorization and reimbursement decisions and attempts to involve them in decision making, raises questions about the limitations of current approaches being used to engage patients in decision making. While it is not the sole intention of these decisions to satisfy patient preferences, the examples noted earlier are all cases of decisions being changed when patients raised their voice, suggesting that they may have been overlooked in the original decision. Meaningful involvement requires that patients are well informed and engaged (Reference deBronkart13;14). Attaining this faces numerous obstacles, including: the complex nature of healthcare decisions, the wide variety of views that patients may have, reluctance to speak up and ask questions, willingness to assent to the recommendations of clinicians (Reference Braddock15;Reference Kravitz and Melnikow16), and lack of presenter skills needed to provide balanced information (Reference Fowler, CA and Sepucha3;Reference Braddock15;Reference Davies and Cleary17;18).
Rather than involving patients in decision making, an alternative approach to incorporating the patient voice is to support decision makers in learning what patients think. This is the motivation behind calls for the quantification of patient preferences. The tendency to focus on aspects of the decision problem that are quantifiable, such as clinical efficacy and safety, means that less attention is given to elements that are less easily quantified, such as patient preferences. It is thus argued that, if patient preferences are to be given the attention they deserve within the decision-making process, it is necessary that they too be quantified (Reference Weernink, Janus and van Til19).
Reviews of methods for eliciting quantitative preferences often distinguish indirect methods, such as matching or choice approaches, from the methods used by multi-criteria decision analysis (MCDA) (Reference Weernink, Janus and van Til19). However, this approach inappropriately categorizes MCDA as an elicitation method, rather than a broader process for integrating preference and other evidence into decision making. An alternative approach, adopted by the recent International Society for Pharmacoeconomics (ISPOR) Taskforce on good practice in MCDA (Reference Marsh, IJzerman and Thokala20), identifies both indirect and direct elicitation methods as potential sources of preferences for inclusion in an MCDA
MCDA provides a way to integrate stakeholder preferences with evidence on intervention performance to inform decision-making. This study considers the lessons from application of MCDA in healthcare for efforts to quantify patient preferences to inform authorization and reimbursement decisions. We begin by defining MCDA and describing how it can help decision makers engage with patients. This is followed by an illustration of how MCDAs have involved patients, focusing on the methods adopted to elicit patient preferences. We conclude with lessons from the MCDA literature for how we might quantify patient preferences.
HOW CAN MCDA SUPPORT DECISION-MAKERS’ ENGAGEMENT WITH PATIENTS?
MCDA is a method for disaggregating a decision into its components and systematically addressing them, often quantitatively, to support decision making (Reference Belton and Stewart21). This is done by: (i) defining the decision context: identifying decision makers and the options to be evaluated; (ii) identifying the criteria to be used to assess the options; (iii) Measuring the performance of each option against the criteria; (iv) eliciting preferences for changes within and between criteria; (v) aggregating performance and preferences into an overall assessment of the value of the options.
Although widely applied outside of healthcare (Reference Dodgson, Spackman and Pearman22), MCDA's value to healthcare decision makers has only recently been realized (Reference Baltussen and Niessen23–Reference Thokala and Duenas26), leading to a sharp increase in publications (Reference Marsh, Lanitis and Neasham25). The German Institute for Quality and Efficiency in Healthcare (IQWiG) (Reference Danner, Hummel and Volz27–Reference Muhlbacher, Bridges and Bethge29) and the EMA (11) are piloting its use; and MCDA has been successfully tested for supporting shared clinician and patient decision making (SDM) (Reference Dolan30).
MCDA provides a framework for breaking down a complex, multi-dimensional health decision into more manageable components and addressing each one in turn (Reference Belton and Stewart21;Reference Marsh, Lanitis and Neasham25). By doing so, it enables patients to think through the problem systematically (Reference Mulley, Trimble and Elwyn31), and this minimizes the use of heuristics as often happens when humans are faced with complex problems (Reference Marewski and Gigerenzer32). MCDA supports patient involvement in decision making by means of: (i) participation in the identification of the criteria to be used; (ii) the synthesis of the evidence on the options’ performance in a transparent matrix facilitates patients’ understanding of an often large body of literature; (iii) explicit elicitation of the patients’ preferences (opinion on how important each criterion is, and how they value changes within a criterion), this also allows for differences in patient preferences and potential patient subgroups to be identified; (iv) aggregation of scores and weights into an overall assessment provides a transparent link between patients’ preferences and decisions.
Patient input for an MCDA can be obtained through workshops, allowing them to learn more about the problem and provide informed judgments. These workshops also allow patients to discuss their experiences, explain their views, and seek clarification. In addition, clinical experts participate in these workshops, allowing them to share their knowledge with patients.
These benefits are confirmed by those who have used MCDAs to incorporate patient preferences into coverage decisions (Reference Airoldi, Morton, Smith and Bevan33). They commonly observe participants enjoying and appreciating the logic of the approach, and patients report MCDA helping to improve the quality of the discussion and supporting them to share their knowledge. Participants noted that it was “an advance on just sitting around a table and talking it through” (Reference Airoldi, Morton, Smith and Bevan33).
MCDA offers other advantages. First, if it improves the transparency of decision-making, it can help patients understand the rationale for decisions. It could also help them appreciate why different decisions are made in different jurisdictions. Second, MCDA generates information that can be used beyond the specific decision at issue. By elucidating how important each criterion is to patients, rather than only obtaining directly their opinion of each option, MCDA generates data that can be applied to other decisions where the same criteria apply.
HOW HAS MCDA BEEN USED TO QUANTIFY PATIENT PREFERENCES?
There are several published studies in which MCDAs incorporate patient's preferences (see for instance Marsh et al. (Reference Marsh, Lanitis and Neasham25). These were undertaken to inform a range of healthcare decisions, including coverage decisions (Reference Hummel, Volz and van Manen28;Reference Airoldi, Morton, Smith and Bevan33–Reference Youngkong, Teerawattananon and Tantivess36) and prescribing determinations (Reference Dolan37–Reference Hummel, Snoek, van Til, van Rossum and Ijzerman39). They involved patients in both the identification of criteria (Reference Airoldi, Morton, Smith and Bevan33;Reference Sussex, Rollet and Garau35;Reference Youngkong, Teerawattananon and Tantivess36;Reference Hummel, Snoek, van Til, van Rossum and Ijzerman39) and also in the quantification of the value of preferences (Reference Hummel, Volz and van Manen28;Reference Goetghebeur, Wagner and Khoury34–Reference Broekhuizen, Groothuis-Oudshoorn, Hauber, Jansen and IJzerman40). The following examples illustrate the diversity of methods used to quantify preferences.
Example 1: Simple Direct Rating Approaches
Direct rating is one of the simpler methods for allocating weights to criteria. The most straightforward approach is to allocate 100 points across the criteria, in proportion to their importance. This can also be done using a visual analogue scale to rate the importance of each criterion on a 100-point scale, or by giving the most important criterion a weight of 100, and allocating points to the other criteria to reflect their importance relative to the most important one.
Sussex et al. (Reference Sussex, Rollet and Garau35) used direct rating in a study designed to pilot the use of MCDA to value orphan medicinal products. Based on an extensive review of the literature and stakeholder engagement, the authors identified the following criteria: availability of treatments, survival prognosis before treatment, morbidity before treatment, social impact of disease on patients’ and caregivers’ daily lives before treatment, treatment innovation, clinical efficacy of treatment, treatment safety, and social impact of treatment on patients’ and caregivers’ daily lives.
Weights for these criteria were elicited during structured workshops with patient representatives. The opinions of clinical and health economics experts were also obtained in a separate workshop. Each workshop comprised 6–11 participants, during which medical specialists provided patients with factual information on the disease and treatments being assessed. Participants within each workshop were divided into small subgroups to assign weights to the criteria. First, they were asked to organize the criteria into groups of “high,” “medium,” or “low” importance. They then were asked to reach a consensus within their groups about how to allocate 100 points among the criteria to reflect their relative importance. Differences between the subgroups’ weights were discussed and resolved so that there was an agreed upon set of weights for each workshop: one set of weights for patient groups, and another set for experts.
The result of the weighting exercise reinforces earlier observations that clinicians’ view of the importance of decision criteria diverge from those of patients. Patient groups gave lower weight to the availability of treatments (11 points out of 100 compared with 19.5 points for clinical experts) and clinical efficacy (17.5 points compared with 27.5 points), and greater weight to the social impact of the disease without treatment (15 points compared with 8 points) and the social impact of treatment on the patients’ and caregivers’ daily lives (17.5 points compared with 11). Although the authors caution that further research should be undertaken to confirm that these results reflect a genuine difference of preference, and not, for instance, a different understanding of the evidence.
The authors concluded that the pilot demonstrates that an MCDA approach is a viable proposition for valuing orphan drugs, and could be used by payers and HTA bodies. They observed that participants in the studies were able and willing to participate in the weighting exercise, and the workshops proved an effective way of achieving a buy-in to a collect weight, suggesting that this type of approach would be relatively simple to incorporate into HTA processes that already involve committees of clinical experts and patient representatives.
Example 2: Pairwise Comparison Approaches
Another weighting method is known as the analytical hierarchy process (AHP), which uses pairwise comparisons of criteria. Participants are asked to indicate the relative importance of two criteria on a scale of 1 to 9, where 1 indicates the criteria are equally important and 9 indicates that one criterion is extremely important relative to the other.
Dolan et al. (Reference Dolan, Boohaker and Allison38) used AHP to determine the weights given by people at average risk for colorectal cancer to criteria for choosing among various screening strategies. The evaluation considered multiple screening options, including: guaiac-based and immunochemical fecal occult blood tests, flexible sigmoidoscopy, double-contrast barium enema, standard colonoscopy, computed tomography colonoscopy, and fecal DNA test. On the basis of American guideline statements, the researchers identified six criteria: ability to prevent cancer, avoidance of side effects, minimizing false positives, and logistical complexity divided into three sub-criteria: frequency of testing, preparation required, and method of testing procedure.
Afterward, the authors asked patients several questions to assess the feasibility of using AHP. A high proportion (92–93 percent across the five sites in which the study was undertaken) of the 484 participants indicated that it was not hard to understand the criteria; most found it easy to follow the pairwise comparison process (91 percent) and make the comparisons (85 percent). The majority (88 percent) stated that they would be willing to use a similar procedure to help make important healthcare decisions. Thus, the authors concluded that it was possible to use AHP to foster patient-centered decision making and that patients are able and willing to perform such a complex MCDA.
LESSONS FROM THE MCDA LITERATURE ON HOW PATIENT PREFERENCES SHOULD BE QUANTIFIED
There is substantial diversity in MCDA approaches (Reference Marsh, Lanitis and Neasham25). Apart from the direct rating and the AHP methods described above, two other weighting techniques warrant mention. First, there are techniques that focus on the range of observed performance on each criterion rather than the criteria themselves (so called “swing” weights). Raters are asked to consider how much the performance of options differs and to weight higher those criteria where the range of performance is more important. Second, rather than eliciting weights directly as in the approaches above, in the discrete choice experiment (DCE) analyses respondents’ choose between hypothetical interventions to infer the weights attributed to the criteria.
There is currently a lack of formal guidance in the MCDA literature on which of these elicitation methods is most appropriate. Several frameworks have been proposed that identify the differences between MCDA methods (see for instance De Montis et al.) (Reference De Montis, De Toro, Droste-Franke, Getzner, Spash and Stagl41). These were not, however, developed for a healthcare audience and will need further translation to elaborate their implications for how to use MCDA to elicit the patient voice. Recent efforts have started to generate such use-specific guidance, including the initial outputs from the International Society for Pharmacoeconomic and Outcomes Research's MCDA Emerging Good Practices Taskforce (Reference Marsh, IJzerman and Thokala20;Reference Thokala, Devlin and Marsh42). This section highlights three factors from this literature that should be considered when eliciting patient preferences.
First, it is important to give appropriate consideration to the heterogeneity of patient preferences. In the example of AHP in the previous section, the authors observed that weights varied widely between respondents. Such variation could have implications for decisions, and it is important that efforts to quantify preferences explore these implications. For instance, differences in preferences may mean that a treatment has a positive benefit-risk balance for one patient or group of patients, but not for others.
This observation calls into question the appropriateness of either using averages of preferences or aiming for a consensus on preferences. There may be circumstances where consensus is an appropriate objective, but it is important that this is justified. For instance, we would argue that the most appropriate approach to quantify patient preferences for consideration by HTA committees would not be by generating a consensus among a small number of patients within a workshop setting (such as the approach adopted by Sussex et al. (Reference Sussex, Rollet and Garau35) in Example 1 above).
Rather, we would recommend that, just as with quantitative measures of treatment efficacy, data on patient preferences should be generated in such a way that differences in preferences can be identified and propagated through the MCDA so that the committee understands the implications of these differences for their decisions. This approach would facilitate a change in the approach to healthcare decision making. Rather than asking whether a health intervention should be made available for a target population, we can ask which patients should be eligible for an intervention given their preferences, although such considerations are less relevant for other decisions. The heterogeneity of preferences is obviously not relevant to SDM, in which a decision is being made by an individual patient.
Second, methods vary in the level of cognitive challenge they pose to participants. This is particularly important to consider when working with patients who may be unfamiliar with the tasks they are being asked to complete. It is important to consider the nature of the elicitation tasks patients are required to undertake. For instance, making a choice (as required by DCE) is perhaps easier than pairwise comparison of the importance of criteria on an ordinal scale (as required by AHP), which in turn is easier than providing a precise estimate of the relative importance of two or more criteria. Elicitation tasks also vary in terms of the amount of information that participants need to consider. For instance, the pairwise comparisons required by AHP are easier for participants than the comparisons required by swing weighting (which need consideration of the range of values a criterion can take) or DCE, which require participants to weigh up options across multiple criteria simultaneously.
Cognitive burden is not just a function of the elicitation task, but also of the patient being asked to undertake the task, and the two should be considered in combination. For instance, the rapidly changing socio-demographics in many countries, including an aging population, mean that there may be more people living with cognitive decline. Equally, some diseases are themselves causes of limited cognitive ability. It is important that researchers consider the population they are engaging, and select an elicitation instrument appropriate to their cognitive abilities, so that these patients too can be included in decision making.
Methods also vary in the level of “support” provided to participants. This is partly a function of whether a workshop or a survey approach is adopted. DCEs typically adopt a survey-based approach, which are inevitably limited in the information they can provide participants and do not allow for interaction with or between participants. DCEs assume that stakeholders possess latent value functions that can be elicited through their responses to surveys. In contrast, other techniques are more often undertaken in a workshop context. This facilitates knowledge sharing between experts and participants, allows participants to clarify the tasks being posed, and also facilitates discussion between participants. The latter is particularly valuable when participants are not expected to have well-formed preferences for the criteria.
Third, it would be wrong to respond to the cognitive burden posed by elicitation method by opting for the least challenging method. Rather, it is important that the cognitive challenges are weighed up against the nature of the results generated by the method. The most commonly applied MCDA methods require that criteria be compensatory, an improvement in one criterion can offset a worsening of another, and that preferences be scaling constants or value trade-offs, reflecting the rate at which changes in criteria compensate one another. It is not immediately obvious that the different methods meet this requirement. Methods often elicit assessment of the relative importance of criteria, rather than formally eliciting trade-offs; that importance is often elicited independent of a range of performance (Reference Keeney43). Swing weighting explicitly considers the range of performance being assessed. But DCE is perhaps the only of our examples where we can confidently say that trade-offs are elicited. This is not to say that trade-offs cannot be elicited with any method other than DCE. Rather, it is important that MCDA is implemented in a manner that helps patients to understand the objective of elicitation task, and how the resulting data will be used. This can be achieved by training patients in the elicitation tasks, piloting the tasks, and validating the interpretation of responses with patients.
CONCLUSION
There is a trend to quantify patient preferences to facilitate their incorporation into decisions. Such elicitation of preferences is an important part of MCDA. MCDA is increasingly being used in healthcare, and there are several examples of it being used to capture the patient voice. Furthermore, the guidance developed to support the use of MCDA in healthcare provides some useful considerations when quantifying patient preferences; namely, researchers should give appropriate consideration to: the heterogeneity of patient preferences, and its relevance to decision makers; the cognitive challenges posed by different elicitation methods; and validity of the results they produce.
CONFLICTS OF INTEREST
Kevin Marsh is employed by a consultancy that received funds for the work performed on this manuscript. Dr. Caro is employed by a consultancy that received funds for the work performed on this manuscript. Dr. Zaiser is employed by a consultancy that received funds for the preparation of this manuscript. Dr. Heywood has nothing to disclose. Dr. Hamed reports personal fees from Sanofi Genzyme, outside the submitted work.
