Many jurisdictions have developed guidance for conducting and interpreting economic evaluation, often in the service of larger health technology assessment (HTA) and reimbursement processes (Reference Tarn and Smith1;Reference Eldessouki and Dix Smith2). However, methodological challenges can arise with specific interventions or disease states. In Canada for example, recent guidance for economic evaluations for oncology products has been developed to address some ambiguities in the general guidance (Reference Mittmann, Evans and Rocchi3). The specific guidance implicitly recognizes the need for more consistent application of economic evaluation as more generic guidance provides opportunities for analysts to use significantly different approaches that in turn could lead to significantly different findings. Methodologic guidance may also enhance the confidence and strengthen the credibility of the use of these analyses by end users (Reference Drummond, Sculpher, Torrance, O'Brien and Stoddart4).
Personalized approaches to medicine can be broadly defined as those using any type of patient-specific information (e.g., genetic and molecular laboratory-based diagnostic, questionnaires, risk scores, patient-reported outcomes, imaging, biometric/functional measures, laboratory-based anatomic pathology, point of care testing, and from-home testing) or patient-specific therapy (e.g., autologous cell therapy). Collecting and creating new information metrics for personalizing therapy to patients has been greatly facilitated by developments in the application of organizational (e.g., databases) and consumer-driven information technology (e.g., cellular phones and home computers). For the purposes of this article and because of its health technology policy relevance, we more narrowly define personalized medicine approaches as “the use of genetic or other biomarker information to improve the safety, effectiveness, and health outcomes of patients by means of more efficiently targeted risk stratification, prevention, and tailored medication and treatment-management approaches” (Reference Faulkner, Annemans and Garrison5).
Diagnostics-based personalized medicine potentially hold the promise to be of benefit to multiple stakeholders, including nations seeking to improve economic opportunities through knowledge- and genome-based applications; drug innovators who aspire to curb expensive development costs (Reference Stallings, Huse and Finkelstein6); healthcare providers who believe safer and more effective care will result from personalized medicine interventions in an aging population with more complex disease; and the health informatics industry which continues to seek health systems applications for “big data.” The use of diagnostic-based information has seen recent growth from these factors as well as incentives for higher levels of coordination and human resource cost pressures, leading to innovation in computed interpretation, automation, and digital reporting (Reference Repa7;8).
With accelerated growth in both the volume and price of diagnostic tests in the last decade, the current and potential economic burden for payers has made this a priority for discussion across the HTA community (Reference Becla, Lunshof and Gurwitz9). Interest in the evaluation of PM can be seen as an extension of previous attempts to standardize evidence-based approaches to assessing diagnostic tests. These include a 2003 International Task Force, (Reference Bossuyt, Reitsma and Bruns10) and the 2004 Evaluation of Genomic Applications in Practice and Prevention (EGAPP) initiative (Reference Teutsch, Bradley and Palomaki11).
The potential to access this new technology, in turn, is driving the need for assessments to support decision making. Like any health intervention, PM interventions have costs and consequences that require consideration by reimbursement authorities who seek to derive maximum value from limited resources. The increased need for evaluation from decision makers can be observed in a 13 percent annual growth in the number of published economic evaluations of diagnostic interventions (Reference Fang, Otero, Greenberg and Neumann12).
Current approaches to economic evaluation to support decision making are largely focused on reimbursement of drugs. Drug reimbursement evaluations are typically population-based, involving single interventions in single populations. Because personalized medicine, by definition, leads to restricted populations or individual care there are questions as to whether current approaches to economic evaluation are adequate for PM interventions. Of specific concern is whether current guidance for economic evaluation is specific enough to avoid inconsistent evaluation and contradictory findings. This, in turn, may lead to inconsistent decisions and may also lower decision-makers’ confidence in the usefulness of economic evaluations to support decision making.
This article addresses the question of whether current guidance for economic evaluation is adequate to support decision making for personalized medicine interventions. It may also be useful to decision makers rely on economic evaluations of PM and those who are updating jurisdiction-specific guidance for economic evaluation or considering the future of health technology assessment and reimbursement policy.
METHODS
The key issues in the economic evaluation of personalized medicine were identified through a workshop and informed discussion by a panel of experts (J.H., A.L., D.M., S.P.). Key issues were believed to be those choices in methods that may cause the most variability in findings unless further specified. Members of the panel were selected based on long-standing expertise in economic evaluation and health technology assessment. The workshop was held in Ottawa, Canada, in January 2012 and sponsored by Health Canada, with 150 participants attending from federal and provincial programs across Canada, as well as academia, and not-for-profit organizations.
The panel was presented with potential issues based on the results of an informal survey and rapid review of the literature. The literature search and selection of review articles was conducted by a single author (D.H.) using PubMed Medline and keywords MeSH terms (“Cost-Benefit Analysis”[MeSH] AND (“review”[Publication Type] OR “review literature as topic”[MeSH Terms] OR “review”[All Fields]) AND “genetic testing/economics”[MeSH]). Given only a purposive sample of issues was required to initiate discussion, and the literature is nascent, a systematic review was not conducted in favor of using a less resource-intensive approach. International subject matter experts (see acknowledgements) were identified from this literature search and contacted to discuss issues identified from the review. A second informal search was conducted using the CADTH Web site to identify recent Canadian economic evaluations of personalized medicine. Authors from these studies (see acknowledgements) were contacted to discuss and review issues identified from the original search and those revealed after discussion with international experts.
Potential issues were categorized according to key methodologic categories in the current Canadian guidelines for the economic evaluation of health technologies (13) and presented to the expert panel. These economic evaluation guidelines are divided into fourteen separate sections prescribing appropriate methods. For example, one section titled “Type of Evaluation” suggests the form of analysis undertaken should be a cost-effectiveness analysis using an outcome that is adjusted for health-related quality of life unless another form can be justified. Each identified issue was mapped onto one or more sections. Panelists were presented with experiences from those conducting economic evaluations in Canada (as per above) and from invited international subject experts of using economic evaluation of personalized medicine for decision making (see acknowledgments). There was also more focused discussion regarding two specific issues. Experts were also invited to comment on potential remedies to the issues identified. In addition to the expert panel, workshop attendees were invited to comment on other issues.
A draft report of the workshop along with a final set of issues was prepared and circulated among the expert panel and wider group of workshop participants for comment. This article represents a summary and elaboration of the workshop findings.
RESULTS
Two recent systematic reviews of economic evaluation of pharmacogenomic tests were identified (Reference Beaulieu, de Denus and Lachaine14;Reference Wong, Carlson, Thariani and Veenstra15). A third un-indexed review of reviews known to one of the authors was also used (Reference Paci and Ibarreta16). The presentations and discussion summary from the workshop are available online (http://ihe.ca/research/knowledge-transfer-initiatives/–methodology-forum/personalized-medicine-research-challenge—health-economics-methodology-1/) and economic evaluations are in a Supplementary File, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000142. The workshop discussion led to the prioritization of six specific issues and potential modifications necessary to existing guidelines that would promote consistent evaluations for decision making (Table 1). These six areas of concern related to how study questions are developed (“Study Question”), how populations are characterized (“Target Population”), how comparators are defined (“Comparators”), how the effectiveness of a PM intervention is evaluated (“Effectiveness”), how outcomes are valued (“Valuing Outcomes”), and how resources are measured and valued (“Resource Use and Costs”). Examples and an elaboration of each of these issues are presented in the next section with a summary, examples and a rationale for each in Table 2.
Table 1. Specific Issues in the Economic Evaluation of Diagnostics-Based Personalized Medicine Based on Canadian Guidelines for Evaluation
Table 2. Rationale, Examples and Elaboration of Specific Issues Identified in the Economic Evaluation of Diagnostics-Based Personalized Medicine
Specific Issues and Potential Solutions
Study Question
First, there was recognition that personalized medicine creates problems for analysts when being asked to frame evaluative questions. Analysts are confronted with either developing a question that is most important for a program-specific decision maker, such as a drug formulary director who is responsible for one part of reimbursement (e.g., a drug rather than a drug and test combination) or for a nonexistent (in Canada) third-party payer administrator that can consider all technologies together. For example, Mittmann and colleagues conducted a trial-based economic evaluation of the effectiveness of cetuximab plus best supportive care versus best supportive care only in patients with advanced colorectal cancer refractory to chemotherapy (Reference Mittmann, Au and Tu17). Through this assessment, they were able to examine the subgroup of almost half of patients who had tested positive for a tumor with a wild-type KRAS mutation. Their findings suggest a lower incremental cost-effectiveness ratio (ICER) associated with treatment of these patients (186,000 versus 300,000 CAD/quality-adjusted life-year (QALY). Despite being a high-quality evaluation, this study does not explicitly address the policy question of the value of providing patients access to testing. Rather, it addresses a clinically-relevant but less-policy-relevant question—namely, the value of treating patients with cetuximab who have positive and negative test findings. A model-based analysis by a different group of investigators in 2012 more directly addresses the question of access to testing. Their analysis reveals performing a test improves health outcomes compared with not performing a test and is more costly if cetuximab is part of the treatment pathway (18).
Target Population
In a similar vein, specification of target population groups within the question asked creates particular challenges for analysts. Personalized medicine interventions may accelerate the evolution and development of clinical treatment pathways. Testing reveals heterogeneity and creates multiple subpopulations which unfold according to the sequence information is gathered. Framing questions around PM interventions that have application in multiple therapeutic areas (e.g., interventions that target a generic immune system pathway) then becomes a challenge. Unlike more conventional HTA, Analysts must make clear whether the target population is understood as the total (still unstratified, e.g., biomarkers not yet measured) population or already “personalized” strata (e.g., individuals with specific previously known biomarker values).
An example of this issue can be seen in two independently conducted analyses intended to examine the value of HER2 testing. In one study, analysts used a patient population of pre- and postmenopausal women diagnosed with breast cancer stratified by breast cancer stage and recurrence (Reference Dranitsaris, Norris, Hanna, O'Malley and Gelmon19). One testing sequence to confirm HER2 positivity was applied to both recurring and newly diagnosed cases. In an alternate analysis, seven testing strategies were used but newly diagnosed and recurrent populations were analyzed together (Reference Dendukuri, Khetani, McIsaac and Brophy20). Both analyses focused solely on testing and assumed therapy would be offered regardless.
Comparators
The rapid evolution of clinical pathways from personalized interventions also makes defining interventions and comparators a challenge. The most common intervention may be no treatment or treatment without the use of specific tests yielding further personal information. Clinical pathways may be ill-defined or in development and decision makers may not be able to make reimbursement decisions that encompass all technologies (e.g., companion diagnostics). Interventions may be access to personalized medicine versus no access, a test versus no test, test and treat versus no test and treat, etc. There may also be debate regarding in what sequence testing should be conducted. Sequencing and uncertainty regarding optimal pathways makes the problem of comparators especially significant in PM interventions.
This is illustrated in an evaluation conducted by Paulden and colleagues examining the impact of introducing a twenty-one-gene recurrence score assay in Ontario (Reference Paulden, Franek, Pham and Krahn21). The analysts enumerated twenty-four clinical pathways from twelve unique risk categories based on two tests with two chemotherapeutic regimens. This resulted in 1,000 unique strategies based on permutations from sequencing test (Reference Paulden, Franek, Pham and Krahn21).
Effectiveness
Another issue for analysts identified is how to best characterize the effectiveness of the intervention. There is general acknowledgement that appropriate data may be largely unavailable as currently regulators do not require proof of clinical efficacy for a test or even sensitivity/specificity which could be used to estimate model effectiveness. Even in cases where reliable information is available, such as companion diagnostic technologies with clinical efficacy information, variable behavioral responses by physicians and patients to personal information and population heterogeneity may limit the ability of analysts to extrapolate across population groups. As with drug therapies, it will be is important to have good information about how likely patients are to comply and adhere to the interventions. However, unlike drug therapies, the issue of compliance in diagnostics-based PM is exaggerated, because there is multiple stages of compliance, including caregiver compliance to ordering a test, patient compliance to receive a test, caregiver compliance to interpret and act on test results, and compliance to subsequent interventions.
Two separately conducted evaluations of use of the twenty-one-gene score assay in a Canadian setting relying on different sources for effectiveness demonstrate this point. They yielded a more than twenty-fold difference in QALYs (1.32 versus 0.05) (Reference Paulden, Franek, Pham and Krahn21;Reference Hannouf, Xie, Brackstone and Zaric22). This issue suggests current recommendations could be strengthened so that compliance and adherence are always accounted for and that synthesis-based estimates must be used in either the base case or sensitivity analysis. This in turn will reduce variability of findings from seemingly arbitrary choices in analytic judgments.
Valuing Outcomes
Another challenge not fully addressed in the guidelines is valuing patient outcomes from an individual versus population-based standpoint. Even if outcomes and their associated heterogeneity are well estimated and defined, it is possible that average population-based valuations of preferences will sharply contrast with the preferences of those who are actually eligible for personalized treatment (i.e., population heterogeneity leads to preference heterogeneity). This difference may lead to a different direction or magnitude in measures of preference-based outcomes and be meaningful from a decision-makers standpoint. Additionally, some personalized medicine interventions are intended to reduce harm while others are intended to augment benefits. There may also be important underlying heterogeneity in preferences (for reduced harm versus improved benefit) from a population and patient standpoint that need to be considered in the analysis.
The importance of this issue and its effect on the findings of economic evaluation should be investigated further. In a modeling study, Basu and Melzer showed that treating individuals for prostate cancer according to individual preferences versus using population-based averages resulted in an additional value of 2,958 USD per patient (assuming an additional QALY is 100,000 USD) (Reference Basu and Meltzer23). However, how this issue should ultimately affect current recommendations is not straightforward. There are additional philosophical questions as to whether this type of approach is even appropriate to support decision making. Until more is known, analysts should consider analyzing the effect of preference heterogeneity based on current information in sensitivity analyses.
Resource Use and Costs
Another issue identified was that of resource valuation (costing). Canadian economic evaluation guidelines suggest using economic (opportunity) costs as the basis for valuing resources and in principle, using total mean cost (including capital and allocated overhead costs) as the unit cost measure (13). Others have noted that the costs of tests may depend on number of tests performed and be difficult to value (Reference Wong, Carlson, Thariani and Veenstra24). In Canada, the cost of testing may be particularly variable due to a large geography and high variability in population density. While some labs may be able to offer high throughput systems and low marginal costs, others may be faced with much higher per-test costs. This issue suggests sensitivity analysis of cost of testing by volume should be performed in every evaluation to better inform payers.
The guidelines also do not currently speak to how to value costs in cost-sharing arrangements between private sector producers and patients. Some companion diagnostics may be offered by a pharmaceutical innovator at “no cost” but analysts must be sure to capture costs because it is actually borne by the third-party payer through the price paid. One remedy would be to develop specific rules for valuing cost-sharing arrangements to avoid variability in findings. This may be particularly important when evaluations are conducted using different funding sources that must cover these costs.
DISCUSSION
The findings presented here suggest that the economic evaluation of personalized medicine may require further specification of approaches than exists in Canadian guidelines. Without specific remedies to the issues identified, there is the real potential for large variations in findings and interpretations, and ultimately, inconsistent decision making. Nevertheless, there was consensus that a new paradigm for evaluation is not required.
The issues identified should not be considered a comprehensive list in the realm of evaluating diagnostics-based personalized medicine nor are our proposed solutions definitive. The approach taken has limitations worth mentioning. First, the majority of the expert panel, workshop attendees were Canadian and the examples used to prioritize the issues were mostly restricted to conducting economic evaluation within the Canadian context. The selection process to identify relevant literature was not systematic, and used a single database and reviewer. This was appropriate as the intent was only to develop a purposive sample of issues and evaluations to inform discussion and not a definitive list. We also did not use a structured method for consensus development. Given these limitations, the issues identified here should be seen as a starting point for further conversation and guideline development.
Despite these limitations, we believe the experience of the panel and the use of reviews to identify potential issues adds strength to our findings. Despite an examination of Canadian experiences, we would also surmise these findings are generalizable to other jurisdictions as they are consistent with other reports outside of Canada using similar methods to identify issues. This includes a recent report from the Institute of Medicine identified several challenges with economic evaluation that we identified (25). Another recent recommendation for evaluating targeted cancer interventions emphasized the need to capture costs and health outcomes in either testing or risk algorithms (Reference Elkin, Marshall and Kulin26;Reference Ferrusi, Leighl, Kulin and Marshall27). Most recently, Annemans and colleagues have similarly suggested standard approaches are sufficient but certain areas require further emphasis (Reference Postma, Boersma and Vandijck28). They identified, as we have, two key challenges are creating research questions and defining comparators when tests are sequenced (Reference Annemans, Redekop and Payne29). They also highlighted the various technical challenges with translating analytic validity into clinical consequences and some of the implications of decision making from the use of economic evaluation.
Unlike our findings, others have questioned the overall value of current frameworks and suggesting they may represent barriers to uptake. Notably, the Personalized Medicine Coalition has asked for more “transparency and predictability” in evaluation and that “different approaches taken by the FDA and (Centers for Medicare and Medicaid Services) in their evaluation of the evidence to cover the cost of the test for patients pose significant challenges to bringing products to market” (30). Similar comments and proposals have arisen from the European Device Manufacturers Association; European Personalized Medicine Diagnostics Association and, in Canada, the Center of Excellence in Personalized Medicine. Issues such as the non-health related value of “knowing” have been used to highlight the limitations of economic evaluation (Reference Lee, Neumann and Rizzo31;Reference Birch and Donaldson32). Additional information from personalized medicine interventions, it has been argued, may lead to improved social welfare from nonhealth decisions related to the interventions (Reference Basu and Meltzer33).
We believe our findings are generalizable to other jurisdictions that use HTA to inform decision making, particularly to publicly funded systems that use economic evaluation to aid resource allocation. This is because many jurisdictions have developed similar guidance for economic evaluation and HTA (Reference Tarn and Smith1). However, even if current approaches to evaluation are adequate, policy makers in other jurisdictions still face challenges when applying the findings from economic evaluations—specifically when faced with decisions on how to fund new drugs that require companion nondrug technology (i.e., diagnostics). In Canada, this was observed when authorities responsible for approving reimbursement of a new and expensive drug (ivacaftor) in an out-patient setting were not responsible for reimbursement of the genetic test required for its indicated use. The lack of integrated HTA and reimbursement structures has been seen in other jurisdictions, notably Australia, which has attempted to merge separate processes for drug and non-drug technology decision making. Planning for the emergence of companion diagnostics required an inevitable coordination between Australia's Medicare Benefits Schedule and Pharmaceutical Benefits Scheme (Reference Merlin, Farah and Schubert34). Co-dependent and hybrid technologies have been granted a single access point to ensure vendors will be steered through to a decision in a timely manner (35). This, in turn, has required underlying coordination of its three HTA advisory committees and methods used for assessment, building on already-extensive guidance on restrictions and the use of diagnostics.
In a similar manner, the parallel evolution of drug and non-drug will require the parallel development of administrative datasets. One evaluation identified revealed significant limitations with the use of administrative data (Reference Elkin, Marshall and Kulin36) and the need for laboratory records that are linked (or can be linked) to other sources of drug performance information. Personalized medicine also underscores the need for additional information, such as patient and physician responses to diagnosis, which are not readily available from clinical trials or administrative data sets. Trialists or those creating observational data sets should consider the value to payers of these factors when planning protocols.
A small but critically important field of research continues to highlight opportunities to identify and evaluate personalized medicine beyond a traditional companion diagnostic paradigm (Reference Basu37). Administrative data have the great potential to allow analyses of patterns of physician and patient response to treatment. However, administrative data may also be inadequate, missing important pieces of information critical for evaluation (Reference Ferrusi, Earle and Trudeau38). Statistical methods can help identify factors that lead to patterns of improved response allowing for better individualized treatment with existing therapies (Reference Basu, Heckman, Navarro-Lozano and Urzua39). This new arm of inquiry suggests we can discover how to achieve improvements on an individual level without the need for developing a companion diagnostic test.
Our findings also highlight areas for future research. This includes empirical exploration of the issues identified by us and others to gauge the sensitivity of results to analytic judgments. It may also include the use of stronger consensus methods to identify issues or create methods manuals such as the UK Diagnostics Assessment Program manual (40). Further development of methods and the need for additional analysis may also have implications for reporting. Another possible avenue of research is to examine the adequacy of the recently published Consolidated Health Economic Evaluation Reporting Standards (CHEERS) for reporting evaluations of personalized medicine (Reference Husereau, Drummond and Petrou41).
CONCLUDING REMARKS
We have described current issues facing analysts conducting economic evaluation to support decision making. The findings suggest there may be opportunities to improve current guidelines for economic evaluation of personalized medicine interventions in Canada. We believe the lessons learned here may also be helpful to addressing shortcoming in other jurisdictional guidelines.
Our findings suggest the economic evaluation of personalized medicine may be more complex, but may not require a new paradigm. Rather, existing approaches and tools used to support analyses can be improved to undertake more complex analyses required for diagnostics-based PM.
SUPPLEMENTARY MATERIAL
Supplementary File: http://dx.doi.org/10.1017/S0266462314000142
CONTACT INFORMATION
Don Husereau, BScPharm, MSc (dhusereau@ihe.ca), Senior Associate, Institute of Health Economics, Edmonton; Adjunct Professor, Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Canada; University for Health Sciences, Medical Informatics and Technology, Hall in Tirol, Austria
Deborah A. Marshall, PhD, Associate Professor, Faculty of Medicine, Department of Community Health Sciences, University of Calgary, 3280 Hospital Dr. NW, Calgary, Alberta, Canada T2N 4Z6
Adrian R. Levy, PhD, Professor, Department of Community Health and Epidemiology, Dalhousie University Faculty of Medicine, 5790 University Ave, Halifax, Nova Scotia, Canada B3H 1V7
Stuart Peacock, PhD, Co-Director, Canadian Centre for Applied Research in Cancer Control, British Columbia Cancer Research Centre; Associate Professor, School of Population and Public Health, University of British Columbia, 2206 East Mall, Vancouver, BC, Canada V6T 1Z3
Jeffrey S. Hoch, PhD, Director, Pharmacoeconomics Research Unit, Cancer Care Ontario, Canadian Centre for Applied Research in Cancer Control, Toronto ON, M5G 2L7, Canada
CONFLICTS OF INTEREST
D. Husereau reports receiving payment for supporting the workshop and drafting the manuscript from the Institute of Health Economics. D. Marshall reports Ad Hoc consulting for Optum Insight on health economics and health outcomes research projects. A. Levy reports receiving travel reimbursement for costs of attending the Symposium in Ottawa in January 2012. S. Peacock has nothing to disclose. J. Hoch has nothing to disclose.
