Health systems around the globe are being asked to make decisions about funding new, high cost medications within constrained budgets (Reference Chalkidou, Lopert and Gerber1). The sheer pace of technological discovery coupled with a growing public demand for innovative therapies means that decisions are frequently being made under pressure on premature clinical data and thus highly uncertain economic evidence (Reference Cleemput, Neyt, Thiry, De Laet and Leys2–Reference Wieseler, McGauran and Kaiser5). Although this is broadly true of current decision making about drugs, pressure on decision making is greatest for diseases which have severe morbidity/mortality consequences (such as aggressive and end-stage cancers) especially when these are rare, because of the limited number of treatment options and limited opportunity for more robust evidence generation (6;7).
Some health systems heavily rely on economic evidence and particularly incremental cost effectiveness for these decisions. Nevertheless, it is recognized that criteria that extend beyond costs, effectiveness, and quality-adjusted life-years (QALYs) are considered by decision makers—whether explicitly or implicitly (Reference Cleemput, Neyt, Thiry, De Laet and Leys2). As a result, there is growing interest in the use of multi-criteria decision analyses (MCDA) to support drug reimbursement decision making (Reference Angelis and Kanavos8). Although MCDA best practice guidelines have been developed (9;10), and the usefulness of a MCDA framework to a drug advisory committee previously assessed (Reference Tony, Wagner, Khoury, Rindress, Papastavros and Oh11), very limited guidance about implementing MCDA into a functioning drug decision-making process exists.
To address this gap, we report the implementation of an MCDA tool incorporated into a Canadian drug reimbursement decision-making process. The paper first provides commentary on drug reimbursement decision-making processes and the rising interest in MCDA. Then, we use a Canadian provincial case study to identify the aspects of the MCDA tool and the context that promoted its uptake into an established Health technology assessment (HTA)-driven drug reimbursement decision-making process based on participant reflections on the process. To our knowledge, no other equivalent report exists for a Canadian drug formulary decision-making context.
The Rise of Multi-Criteria Decision Analysis (MCDA) in Drug Reimbursement Decision Making
Sharp increases in the price of new pharmaceuticals alongside growing pharmaceutical expenditure have forced an acute focus on the processes that inform drug reimbursement decision making. This is particularly relevant in settings where pharmaceuticals are publicly financed and decisions about funding (and not funding) a medication are a part of a political process and so subjected to intense public scrutiny. Consistency, transparency, and fairness in resource allocation decisions are unsurprisingly demanded in these settings (1;12).
HTA is used to inform drug reimbursement decisions across many jurisdictions (Reference Banta13). HTA assesses evidence about the clinical effectiveness, cost effectiveness, and budget impact of pharmaceuticals being considered for adoption. Until recently, the focus of HTA has largely been on adoption of new technologies, but this is starting to shift as the concept of health technology management has emerged and more focus is being given to disinvestment and reassessment activities (14;15). There has also been a movement toward incorporating additional criteria in HTA such as equity, access, and severity (Reference Cleemput, Neyt, Thiry, De Laet and Leys2), it is the output of cost-effectiveness analyses—that is, the incremental cost effectiveness ratio (ICER)—that is often used as a focal point for decision making. In some contexts, this may only be to negotiate lower prices (Reference Franken, Heintz, Gerber-Grote and Raftery16).
In various markets such as the United Kingdom, Australia, and Canada, the ICER is benchmarked against an “ICER threshold” which historically was arbitrarily set (Reference Birch and Gafni17), though methods to more rigorously determine the threshold exist (18;19). The ICER threshold is said to represent the extent of society's willingness to pay for an improvement in an additional year in full health through the use of a drug compared to its relevant comparator. Moreover, application of the ICER threshold is typically applied to one-off decisions about individual medications (as opposed to whole of class) (Reference Gafni and Birch20).
However, as the ability to pay for medications is stretched both at the health system and individual levels, valid questions about the current approach to drug allocation decisions have emerged. Akin with the warnings of health economists three or more decades ago, one critical issue with the ICER approach just described is that decisions can be made without information about the opportunity cost of a reimbursement decision across a health portfolio (17;20;21) (though methods in response to this issue have recently been developed (18;19)). Without this understanding, decision makers cannot assess whether the (often large) shifts in budget allocation that are required to pay for a new innovative medication provides overall greater benefits than the current portfolio of investments. Furthermore, identifying opportunities for disinvestment only happens passively and is limited to clinical no-brainers (e.g., where older drugs are no longer in use because of newer, safer, and more effective alternatives) and quick wins. Without an overall budget allocation strategy, in the absence of an increase to budget, the only stop-valves to public expenditure are to shift costs onto patients (in the form of out-of-pocket payments) or third-party payers, or to delay or deny publicly funded-access to new innovative medications. Clearly each of these scenarios threatens the notion of equitable access to medications across a population.
In addition to not addressing overall affordability, assessing the value of a medication only according to the ICER or to cost per QALY, has been found inadequate—especially given the importance of other key criteria such as equity that are considered by decision makers (Reference Cleemput, Neyt, Thiry, De Laet and Leys22). Concerns remain even as staunch supporters of QALY approaches advocate for adjusting the QALY with weights or using different ICER thresholds to reflect these other criteria (Reference Bobinac, van Exel, Rutten and Brouwer23).
MCDA has been proposed as an alternative. MCDA, as the name suggests, is a method for assessing the value of a proposal based on multiple criteria that are considered relevant to a decision (Reference Baltussen and Niessen24). Each criterion is assigned a weight, and each proposal is evidence-rated against the criterion to produce a “benefit score.” A proposal's scores for each criterion and for the collection of criteria can then be used as a catalyst for deliberation in a reimbursement decision (Reference Baltussen, Marsh, Thokala, Diaby, Castro and Cleemput25). The approach has been used for many years in non-drug contexts (Reference Baltussen, Mikkelsen, Tromp, Hurtig, Byskov and Olsen26–Reference Youngkong, Teerawattananon, Tantivess and Baltussen28) and recently, best practice methods for healthcare decision making have been issued (9;10). Contemporary academic debate is focused on how best to weight and score criteria, whose values should be elicited in the establishment of criteria and how budgetary concerns should be incorporated into the MCDA (8–10;29;30).
Attempts to justify the robustness of the approach compared to current HTA processes have also been made (Reference Kolasa, Zwolinski, Kalo and Hermanowski31). Although this is important, it overlooks the fact that MCDA has been widely applied outside of the heath sector as a robust tool to support decision making and the output of MCDA does not need to be the decision-point for resource allocation nor replace a deliberative process (25;32). Rather, MCDA should be a part of the broader domain of priority setting and resource allocation decisions where considerations of budget and opportunity cost occur concurrently. As such it may be helpful to distinguish between “qualitative” MCDA and “quantitative” MCDA (Reference Baltussen, Marsh, Thokala, Diaby, Castro and Cleemput25). The former is described as a deliberative process that embeds the use of MCDA within larger discussions about overall affordability and value for money relative to the broader portfolio of investments. For the latter, MCDA is viewed as a tool that produces an output that is in essence used as the final decision-making product.
Importantly, elements of an MCDA approach have already been adopted by various HTA agencies across the globe (e.g., Quebec's Institut national d'excellence en sante et an service sociaux (INESS)) (33). To some extent, MCDA is also being used in other jurisdictions but with insufficient explicitness or transparency about the framework or the weight of the criteria being considered in the decision process (e.g., New Zealand and Australia). With the MCDA ship already sailing around the globe, we suggest the important question is not whether MCDA should be used, but how best to implement MCDA into already existing drug reimbursement processes. To date, some progress has been made on describing real-world application in drug decision-making contexts (31;34)—this paper seeks to build on this previous work in describing a case study from Canada.
Case Study: Implementing MCDA into Drug Decision Making in a Canadian Province
Background and Journey
In Canada, publicly financed, out-of-hospital medications are primarily provided via provincial and territorial governments. Drug reimbursement decisions are thus made at the provincial level. However, as part of the national Common Drug Review process, recommendations for listing are provided by the Canadian Drug Expert Committee (CDEC) to the provinces based on HTA evidence. The provinces are not in any way obligated to agree with CDEC recommendations (Reference Allen, Walker, Liberti and Salek35).
In our Canadian provincial case study, decisions to include a medication on its publicly funded drug program are made by the Minister of Health. To inform this decision, an independent advisory body (IAB)—provides evidence-informed recommendations about the potential value of subsidizing candidate medications using public funds. The IAB is comprised of both professional members with expertise in critical appraisal, medicine, ethics, pharmacy, and health economics as well as public members. The IAB recommendations are not necessarily followed, although the vast majority of IAB recommendations are taken up. Members typically are appointed to the committee for a 3-year term and may be re-appointed once.
Until recently, the determination of value within this process has fallen broadly within an HTA framework, with the primary levers for reimbursement recommendations being clinical effectiveness and cost-effectiveness. No explicit cost-effective threshold was used, although the package of information that the IAB receives in its assessment often refers to a threshold. Until recently, there has been no disinvestment activity at the IAB level (i.e., the process has focused primarily on adoption).
In 2016, with a fixed budget, the Ministry of Health was faced with finding the resources to fund a pipeline of expensive but potentially valuable medications. There was recognition amongst the Ministry of Health and the IAB that the process then in place was not taking into consideration the opportunity cost of the new technology against the broader portfolio sufficiently well and decisions were being delayed on the grounds of affordability. Accordingly, the Ministry of Health re-assessed the decision-making process. The Ministry of Health sought advice from within the committee about alternative, values-based processes that could promote consistency and transparency around decision making. Although no concern was raised from the Ministry of Health about the consistency between historical IAB recommendations, IAB members were also seeking a robust process that could ensure the systematic incorporation of the public voice into decisions and could consistently capture the broad range of (non-QALY) values that underpinned their recommendations.
Initially, information about MCDA and other priority setting and resource allocation frameworks were presented by the ethicist and health economist to key Ministry of Health staff and IAB members. These interactions highlighted the economic rationale underpinning current pharmaceutical decision making, including the use of QALYs and ICER thresholds, and what MCDA was by comparison. Of relevance was the presentation of practical local examples that highlighted the use, advantages, and disadvantages of the various approaches. For MCDA specifically, a key consideration was how it could eventually feed into the broader evaluation of overall budget spent and resource allocation within the drug plan portfolio.
The MCDA process was considered to be highly compatible with the approach already being employed and, when used as an analytical tool, was thought to formalize the deliberations that were in many ways already taking place. This aligned well with the understanding of using “qualitative” MCDA as touched on above. In this regard, there was no perceived change required to the information received from CDEC in order to use the MCDA. Instead, it was thought that MCDA could make better use of all available information with the added benefit that consideration of this information would be consistently captured between submissions. In addition, it was thought that MCDA conferred the ability to form a record of the type of information that is needed for decision making that may be missing from national submissions, whereas also providing an audit or accountability trail. This latter point was thought to be especially important for transparency. However, perhaps because of the compatibility with the current process, the ability of MCDA—as a stand-alone intervention—to address issues around affordability and budget constraints was not immediately clear. Furthermore, the compatibility of decisions made with and without an MCDA-based process was not known.
Description of New Process
In 2017, the IAB moved forward with implementation of an MCDA-based process and a revised method of deliberation. Development of the new approach was led by the health economist (CM) and health ethicist (BJ) on the IAB, in conjunction with the IAB chair (RC) and Ministry of Health personnel. In addition, an external consultant with MCDA expertise was utilized. A post-doctoral fellow (TL) was invited to observe the process at various points and provide reflections on the workings of the IAB.
The decision to implement MCDA was ultimately made by the full IAB with guidance from the Ministry of Health. Although the committee acknowledged similar moves away from ICER-based thresholds in other jurisdictions, the decision to implement the MCDA process was fundamentally driven by the desire to provide recommendations that were generated from a consistent and transparent, values-based process that could stand-up to public scrutiny over time. In particular, the IAB wanted to move toward an approach that utilized a set of criteria reflective of stakeholder values. A brief description of the process is provided here.
Through extensive committee deliberation over a period of several months, the IAB adopted six criteria: clinical effectiveness; quality of life; safety; severity; unmet clinical need; and equity (see Table 1). The IAB landed on these criteria following a review of criteria used elsewhere, consideration of what was thought to be key values embedded in the given health system context noting that formal public consultation was not undertaken, and input from Ministry of Health personnel. The IAB spent considerable time working on the definitions of the criteria to ensure consistency in rating and then used a point allocation method to weight the criteria that involved each individual allocating points across the criteria and then the committee as a whole reflecting on individual weights and coming to consensus. A formal scoring tool was also developed using a four-point rating scale. An overall benefit score for a given drug is calculated by multiplying the weight by the score for each criterion and then summing across the criteria. This simple scoring approach was endorsed by the IAB and the Ministry of Health.
Table 1. Definitions of criteria
The main thing to emphasize is that the MCDA scoring and overall benefit score output informs a rich, deliberative discussion by the committee for each drug under consideration. Cost per patient as well as overall budget impact is brought into the discussion in parallel to the overall benefit score. There is no attempt to calculate a “value-cost score,” but cost is considered in so far as a low benefit score at a high per patient cost would unlikely receive a positive review. In addition, the committee considers the opportunity cost of the overall spend for the given drug which further contributes to the assessment and value placed on the drug under review relative to other spending priorities. For each drug, the IAB Chair assigns two primary reviewers who lead the initial discussion, one clinical member and one public member. Questions are then posed by the other committee members and detailed discussion ensues. A final “score” for each criterion is reach by consensus, leading to calculation of an agreed overall benefit score. Most of the information related to criteria assessment is included in the CDEC package, which includes a systematic review of the clinical evidence, detailed independent evaluation of the manufacturer economic analysis and the CDEC recommendation. In addition committee members are also free to draw on their own (broader) expertise in the discussion. Provincial level budget impact is completed by the Ministry of Health and included in the package brought to the IAB. Additional expert clinical review (typically two external reviewers) is also included in the package. Patient input is included in the CDEC submission whereas public input is brought to the IAB through membership on the committee itself.
Implementation occurred after several months of engagement with all IAB members in developing, refining, and ultimately piloting the revised approach. Criteria were adjusted to enable consistent interpretation between members and to ensure the criteria were adequately capturing the values that were being deliberated upon. As highlighted above, the deliberation method enables in-depth review of the evidence against each of the criteria and ensures that the voices of all IAB members, regardless of background, were heard and their arguments given due consideration in the deliberation. No changes to policies were required for implementation and the IAB committed to evaluation of the approach over time. The new process has now been used to systematically assess the package of evidence received from CDEC to support new drug submissions since March 2017. Moving forward the MCDA process may also be applied to drugs across the whole drug plan portfolio to inform broader discussions about the opportunity cost of new and existing agents, including the potential for delisting of low value medicines.
Key Observations
The following observations are offered not as part of a formal evaluation (which is touched on below as a potential future step) but rather from the perspective of the authors to stimulate debate noting the early stage of real-world adoption of MCDA in drug decision-making contexts. Most importantly, in our view this case study demonstrates that MCDA can be implemented as part of a broader deliberative process into a functioning drug reimbursement setting. Implementation of the approach was rapid and leveraged the strong formal and informal communication networks already established between the IAB members and between the IAB and the Ministry of Health. As a small committee with long-standing professional relationships, the IAB appeared to have a shared vision for changing to a system that better reflected the (non-QALY) values and deliberations that underpinned their recommendations. This vision was championed by specific members of the committee, specifically the ethicist and health economist, yet the whole committee contributed to the decision to adopt, pilot, and implement the new process.
A clear driver for implementing MCDA in this case study was the affordability challenge faced by the Ministry of Health and the risk of potential delays in medicine approvals. Such delays would impact access to valuable medicines within the community thus opposing the central objectives of the publicly financed drug program. The IAB was also motivated to implement MCDA to meet the needs of the Ministry of Health for consistent and transparent recommendations that could appropriately capture the views of the public who ultimately finance the drug plan. Although there were no explicit complaints on these grounds about IAB recommendations raised by the Ministry of Health, the idea that MCDA could add value to the process in terms of greater accountability, transparency, and the incorporation of the broader non-QALY values that the IAB deliberated upon likely contributed to this positive climate for change.
The position of MCDA set within a deliberative process as a framework for IAB decision making rather than as a replacement of the decision-making process is important. This contrasts with other proposed applications of MCDA where the overall score arrived at through the MCDA lens becomes the decision point (Reference Baltussen, Marsh, Thokala, Diaby, Castro and Cleemput25). Again, we refer here to the helpful distinction between qualitative and quantitative MCDA (Reference Baltussen, Marsh, Thokala, Diaby, Castro and Cleemput25). The use of the MCDA as an input into and indeed as part of the deliberation likely facilitated implementation for many reasons. First, because the criteria were the values already explicitly and implicitly deliberated upon by the committee, MCDA could integrate easily and rapidly into the existing process without the need to change policies and with minimal change to meeting procedures. Second, criteria assessment could be readily applied to the package of HTA-evidence supporting new drug submissions as much of the information already was included in the CDEC package. Finally, because the decision does not rest solely on the precision of the score, the MCDA scoring could be readily interpreted by committee members (Reference Tony, Wagner, Khoury, Rindress, Papastavros and Oh11) and implemented before evidence comparing MCDA to the former non-MCDA, ICER-based approach emerges. All stakeholders have viewed the change as a healthy advance that is still a work in progress.
In addition to viewing the MCDA as part of a deliberative process, it is also being used as an intermediate step toward addressing the fiscal constraints that were catalyzing the review of the previous decision-making process. This is because there is presently no single agreed upon approach to best incorporate budget impact and address issues of opportunity cost when using MCDA (Reference Baltussen, Marsh, Thokala, Diaby, Castro and Cleemput25). However, rather than delay change until academic debates are finalized, implementation was possible because MCDA was not viewed as an all-encompassing solution to the problem. This approach builds further adaptability into the process to emerging research evidence whereas addressing the current concerns of the IAB. Given the real-time fiscal constraints and impending delays in medication decisions faced by the Ministry of Health, implementing an “imperfect academic” tool first and evaluating later reflects a practical solution to a real-world problem.
Future Plans
We have reported on participant reflections on the process rather than conclusions drawn from more regular formal evaluations. Thus, an important part of the implementation of the newly adopted approach to decision making in the IAB will be in its evaluation. This is not straightforward given there are no firm criteria for defining “better decision making” for drug reimbursement decisions. Nevertheless, key indicators of effectiveness for this particular case study could be whether in the long term the approach (i) enables better quality rationale for drug-listing recommendations, (ii) is used to address the affordability issues faced by the Ministry of Health (e.g., via disinvestment decisions), and (iii) remains in use by successive committees and for the latter, the recent evaluation of MCDA in improving consistency and transparency in non-drug HTA will be useful (Reference Kelley, Egan, Stockley and Johnson36). In the short term, further reflections from the committee members on the use and acceptability of the process is needed and will further promote shared learning among drug reimbursement communities.
Although organizational and contextual issues clearly influenced the implementation of qualitative MCDA in this Canadian province, further work is needed to understand whether their presence or absence facilitates or impedes implementation of MCDA in other settings. A key feature driving change was a budget that was truly limited; the extent to which systems with more flexible budgets are equally motivated to review and change their processes is unclear. Nevertheless, given the dearth of literature describing the implementation of new drug reimbursement processes, this paper serves as a foundation for comparing the implementation of MCDA and other methods across Canada and other health systems.
Acknowledgments
This work was supported by the National Health and medical Research Council (TL APP 1110230).
Conflict of interest
The authors have nothing to disclose.
