Economic evaluations of health technologies aim to maximize the societal benefit of scarce healthcare resources (Reference Drummond, Stoddart and Torrance11). Given the increasing demands on healthcare systems and progressively greater emphasis on constraint in spending (Reference Gibson, Martin and Singer15;Reference Mitton, Patten and Waldner25), many health management organizations and individual institutions have turned to formal, evidence-based approaches to inform expenditure decisions (Reference Claxton7;Reference Mitton, Patten and Waldner25). Decision makers require analytic tools to aid in the decision of implementing a health technologies based on the evidence available. Cost-effectiveness analysis has been used as a measure of a technology's value in terms of the health gains it provides; however, there is often a great deal of uncertainty regarding the cost-effectiveness of an intervention, thus clouding the adoption decision. It has been argued that this uncertainty should not affect the adoption decision, however, it can be used to determine whether further research on the effectiveness of an intervention is required (Reference Claxton7). Value of Information analysis has the potential to assist policy makers, researchers, and health research funding organizations in formulating future research programs and to make the best use of limited research funding.
To illustrate the use of value of Information analysis, we present a novel, nonpharmacological health intervention that aims to improve the sensitivity of detection of acute coronary syndrome (ACS) in individuals presenting to emergency departments with complaints of chest discomfort. An estimated 500,000 patients in Canada and 6 million patients in the United States present to emergency departments with a complaint of chest discomfort every year (20;Reference Selker, Beshansky and Griffith29). The majority of these patients do not have ACS and yet over 50 percent of these patients in Canada and the United States are admitted to hospital for further investigation. Less than half of these admitted patients prove to have ACS (Reference Gibler, Hoekstra and Weaver14); the cost to the healthcare system of investigating and treating all these patients who have noncardiac causes for their chest pain is significant. Furthermore, unnecessarily long emergency department stays contribute to the high levels of congestion experienced in Canadian emergency departments in urban centers. Despite the fact that most patients admitted to hospital with chest pain do not have ACS, 2–5 percent of patients discharged home after a short emergency department evaluation prove to have ACS and run the risk of higher mortality and morbidity (Reference Christenson, Innes and McKnight5;Reference Gibler, Hoekstra and Weaver14). The mortality rate for those patients with ACS who are discharged from the emergency department is almost double that of patients with ACS who are admitted to hospital (Reference Pope, Aufderheide and Ruthazer26).
Several risk stratification tools have been developed to assist physicians in identifying those patients with chest pain that are at high risk of having ACS (12;16;17;21;23;27;28;30;31). However, none of these tools help clinicians identify patients that can be discharged safely after a short emergency department evaluation. The “Early disposition prediction rule for patients with chest discomfort” is a research project currently under way in Vancouver, British Columbia, Canada. This project has developed, but not yet validated, a clinical prediction rule that identifies patients who are safe to send home after 2 hours of observation and investigation and within 3 hours of emergency department admission. The derived Early Disposition Prediction Rule has a sensitivity of 98.8 percent and a specificity of 32.5 percent for predicting ACS events occurring within 30 days (Reference Christenson, Innes and McKnight6).
Despite the potential benefits of implementing this clinical prediction rule in emergency departments across the country, there exists a great deal of uncertainty around both cost and outcome parameters in determining effectiveness and cost-effectiveness. Prior to thought of implementing this intervention, this uncertainty clouds the decision to further research the efficacy and effectiveness of the intervention in the form of a validation study. The objective of our study is to demonstrate the utility of Value of Information analysis in aiding the decision to conduct further research, using the example of a newly developed Early Disposition Prediction Rule (EDPR) for patients presenting to emergency departments with chest discomfort.
METHODS
A decision analytic model was constructed to investigate the potential cost-effectiveness of the EDPR and assess the value in acquiring further research to validate the rule (Figure 1). Our study was based on resource utilization and outcome data collected for a previously published study on the development of the EDPR for patients with chest discomfort (5,6), which compared the implementation of the EDPR with clinical judgment alone (hereafter referred to as “usual care”). The usual care cohort was modeled on 1,799 patients who presented with undifferentiated chest pain to the emergency department of two large teaching hospitals in Vancouver, British Columbia, Canada, during the period June 2000 to April 2001 (Reference Christenson, Innes and McKnight5). The EDPR cohort was modeled on the 769 patients used to develop the rule (Reference Christenson, Innes and McKnight6). A total of 221 patients were included in both patient groups and were excluded from our model, leaving 1,578 and 548 patients in the usual care and EDPR cohorts, respectively. Patient characteristics and outcomes for the two cohorts are outlined in Table 1. The time horizon of the study was 30 days after presentation to the emergency department with chest pain.
Figure 1. Decision analytic model. ACS, acute coronary syndrome.
Table 1. Patient characteristics
a Adverse event defined as one or more of the following: cardiac arrhythmia requiring medical intervention, respiratory failure requiring assisted ventilation, proven pulmonary thromboembolism, proven symptomatic aortic aneurysm or dissection, new congestive heart failure requiring intravenous drug treatment, hypotension requiring intra-aortic balloon pump, intravenous drugs or blood transfusion, chest compressions, coronary artery bypass graft), any percutaneous coronary intervention).
AMI, acute myocardial infarction; ACS, acute coronary syndrome; NA, not available, data not collected.
Parameter Estimates: Outcome Probability and Cost Measures
The probabilities of arriving at each decision node in the model were based on the real patient dispositions in the usual care cohort and on the predicted patient disposition if the EDPR had been applied to the EDPR cohort. Patient outcomes (ACS positive or negative) were based on actual patient data obtained from the 2 study populations. ACS was defined as either acute myocardial infarction or definite unstable angina (Reference Christenson, Innes and McKnight5). Our measure of effectiveness was the number of ACS discharges from the emergency department prevented.
Patient-specific resource utilization data for a 30-day follow-up period from index visit was used to calculate costs. Because costs in the EDPR cohort may have been influenced by the development of the rule, costs for both cohorts were derived from the usual care cohort. The costs of hospitalization (Reference Anis, Sun and Singh1;Reference Drummond, Stoddart and Torrance11), surgical procedures (4;Reference Thomas, Nosyk and Fisher33), physician visits (4), and diagnostic tests (4;24) were all included. The analysis adopted the perspective of the Ministry of Health. All costs were reported in 2003 Canadian dollars ($1CAD = $0.81US). Neither costs nor benefits were discounted due to the short duration of patient follow-up.
Analytic Methods: Incremental Cost-Effectiveness
The incremental cost-effectiveness ratio (ICER) was calculated as the difference in mean cost (C) divided by the difference in mean effectiveness (E) between the EDPR and the usual care groups:
Let λ be the maximum willingness to pay for a unit of health benefit—in our case, the prevention of an inappropriate ACS discharge; if ICER ≤ λ, then the CPR can be considered cost-effective. We used probabilistic sensitivity analysis to evaluate the uncertainty surrounding the ICER. This procedure includes two steps: First, specifying distributions for uncertain model parameters in terms of their mean, standard deviation, or other statistics, and, second, using Monte Carlo simulation to select values at random from these distributions to assess the overall distribution of the cost-effectiveness ratio and construct the desired credibility interval (Reference Briggs2;Reference Doubilet, Begg and Weinstein10). We assigned Dirichlet distributions for the parameters, indicating the probabilities of arriving at each of the nodes of the decision analytic model, and normal distributions for all the cost parameters (Reference Briggs, Ades and Price3). Estimates and distributions of probabilities and costs can be found online in Supplementary Tables 1 and 2 (which can be found at http://www.journals.cambrige.org/jid_thc).
Analytic Methods: Expected Value of Perfect Information
Uncertainty surrounding the mean estimate of cost-effectiveness can be costly if it increases the possibility of making the wrong decision in terms of implementing or not implementing the health intervention. The information gained from further research is valuable, as it reduces the expected costs of this uncertainty. The Expected Value of Perfect Information (EVPI) can be interpreted as the expected costs of uncertainty, as perfect information would eliminate completely the possibility of making the wrong decision.
The EVPI is calculated as the difference between the expected value of benefit if the decision is made when the “true” parameter values are known (in other words, with “perfect” information), and the expected value of benefit if the decision is made with uncertainty in all parameter values (current information), as follows:
where Ba represents the net benefit of a given strategy “a” (in monetary terms), which is a function of θ, the uncertain parameters, and λ, the threshold cost-effectiveness ratio. This net benefit is simply a transformation of the above ICER in monetary terms for a single strategy a, taking into account the threshold ratio, λ:
Therefore, the first term in Eq. 2, the expected net benefit of a decision taken with perfect information, is the mean of the maximum values of net benefit for each iteration of the Monte Carlo simulation, while the second term, the expected net benefit of a decision taken with current information, is the maximum of the mean values of net benefit for each iteration. This calculation is then repeated for a range of values of λ. The EVPI represents the maximum sum that the healthcare system should be willing to pay for reducing the uncertainty in the decision to implement the EDPR. Although the estimated EVPI is the expected maximum value for additional information to inform the treatment of a single patient, the information acquired can be used to treat all patients that may benefit from the intervention in question. We can, therefore, estimate the population or total EVPI by multiplying the per-person EVPI by the number of persons that may benefit from intervention each year, as follows:
RESULTS
The results of our probabilistic decision analytic model suggest that application of the EDPR could result in a cost-savings of $36,564 and 12.2 fewer ACS discharges per 1,000 patients. The cost-effectiveness of the EDPR compared with usual care was −$2,999 per ACS discharge prevented in the base-case analysis, indicating a cost-savings. Probabilistic sensitivity analysis supported the baseline results with a 55 percent probability that the ICER would be negative due to cost-savings. However, the credibility interval surrounding the mean estimate was large, with an ICER of $89,318 at the upper bound of the 95 percent credibility interval. The cost-effectiveness acceptability curve plotted in Supplementary Figure 1 (which can be found online at http://www.journals.cambrige.org/jid_thc) indicates that at a cost-effectiveness threshold ratio of λ = $20,000 per inappropriate discharge prevented, there is a 20 percent probability that the intervention would not be cost-effective, given current information. The threshold ratio of $20,000 per inappropriate discharge prevented represents the approximate opportunity cost of inappropriately discharging then re-admitting an individual who is ACS positive. This opportunity cost does not take into account any incremental loss of quality of life or other health burden, thus the threshold ratio of $20,000 is a conservative estimate of societal willingness to pay.
The EVPI calculations for the decision are presented in Supplementary Figure 2 (which can be found online at http://www.journals.cambrige.org/jid_thc). If we assume λ = $20,000 per inappropriate discharge prevented, the EVPI per patient is $32.59. Alternatively, for a λ = $50,000 per inappropriate discharge prevented, the EVPI per patient is $11.81.
Using a discount rate of r = 3 percent, time horizons of T = 1, 2, and 5 years, and an annual population of n = 500,000, corresponding to the number of individuals presenting to emergency departments with chest discomfort each year in Canada (20), we found a total EVPI of $16.3 million within the first year of implementation, assuming λ = $20,000 (Figure 2). Varying the annual incidence of presentation to ED with chest discomfort by 25 percent in each direction indicates a range of $12.2 to $20.4 million. The total EVPI remained positive even at very high threshold values. At a threshold of $100,000 per inappropriate discharge prevented, tEVPI after 1 year of implementation was $3.5 million. EVPI was higher when considering longer periods of implementation.
Figure 2. Expected value of perfect information for the population of Canada.
These estimates of tEVPI can be used to decide whether to collect more information to inform the decision of implementing the intervention. For different values of λ, tEVPI is compared against the cost of collecting further information in the form of experimental or observational research. For the intervention in question, for λ = $20,000, we have found that the value of further research is $16.3 million in the first year of implementation, thus conducting any research costing more than this would not be cost-effective.
DISCUSSION
In this article, we have demonstrated the utility of Value of Information analysis in guiding decisions to adopt new interventions and to pursue further research. In our example, preliminary analysis suggests that the EDPR may be cost-effective, that is, cost-saving. However, given the small sample size on which our analysis is based and the rarity of the outcome measure of inappropriate discharges of ACS, we conclude that it would be cost-effective to conduct further research to reduce the uncertainty regarding the decision to implement the EDPR on a national scale.
Our example of the decision to implement the EDPR in Canadian emergency departments provides a good illustration of the utility of Value of Information analysis for two reasons. First, the intervention in question is potentially highly cost-effective, if not cost-saving; however, given that the outcome in question is rare and the sample size of the initial observational study is small, there is a high degree of variability in our estimate of cost-effectiveness. If the intervention appeared not to be cost-effective after initial evaluation, further research would have been much more difficult to justify. Second, our decision analysis and the intervention in question present a novel application of Value of Information analysis. Past studies have been used to provide a rational basis with which to inform formulary decisions for Health Management Organizations, such as the National Institute for Health and Clinical Excellence, on whether or not to list a pharmacological intervention on a formulary or to request further research to inform the decision (Reference Claxton, Sculpher and Drummond8;Reference Dong, Coyle and Buxton9;Reference Griebsch, Knowles and Brown18). In our case, we show that this methodology may be used to inform investigators and health research funding organizations on whether to pursue and fund further research for promising nonpharmacological interventions as well.
The need for evidence-based decision making in allocating scarce health resources has never been greater, especially in healthcare systems providing universal coverage. Cost-effectiveness analysis provides decision makers with an analytic tool with which to determine the relative and absolute value of health interventions to provide an ordinal ranking of new interventions in terms of their cost-effectiveness or value for money. Value of Information analysis is an extension of cost-effectiveness analysis which, when used together, can inform policy makers not only in the decision to implement an intervention, but also the decision of whether further research is required before implementation. Further detail on this and related methods can be found in Claxton (Reference Claxton7) and Groot Koerkamp et al. (Reference Groot, Hunink and Stijnen19).
A criticism of EVPI analysis, and economic evaluation of health interventions in general, is that λ, the threshold ICER, is unknown (Reference Gafni and Birch13). The critical value λ is meant to represent the threshold at which decision makers are indifferent between funding the intervention and using the resources in other programs or interventions. Threshold values of $20,000 to $50,000 per quality-adjusted life-year gained have been used (Reference Laupacis, Feeny and Detsky22;Reference Ubel, Hirth and Chernew34), but these estimates have been criticized as being unsubstantiated. As generic measures of health related quality of life were not used in the observational study on which we base our analysis, we use a nongeneric measure of effectiveness, making the choice in threshold value somewhat more difficult. Despite these drawbacks, some have argued in favor of an approach that makes the choice of λ explicit, thus requiring decision makers to contemplate the implicit trade-off they make in choosing which health interventions to implement (Reference Stinnett and Mullahy32). Furthermore, in presenting the results of EVPI analysis, the tEVPI is presented for a range of values of λ, thus presenting decision makers with a range of threshold values for which the decision to conduct further research may be cost-effective.
There were some limitations with our analysis. We were limited to applying the rule to the same cohort of patients that the rule was derived from; the EDPR has not been prospectively applied to a cohort of patients. In our decision analysis, we estimated average costs for each branch of our decision tree from the Usual Care cohort because of incomplete cost information obtained from the Clinical Prediction Rule cohort. The variability in these parameter estimates introduced a degree of uncertainty around our base-case ICER.
POLICY IMPLICATIONS
Our study demonstrates how cost-effectiveness and Value of Information analysis can be applied in the decision to conduct further research on a new health technology. Our results suggest that the health and monetary benefits of conducting further research into the EDPR are likely to outweigh the costs of conducting this research, even in the short-term. Given that approximately 500,000 patients present with chest discomfort to Canadian emergency departments each year, there is potential for a cost-savings of $18.3 million with 6,000 fewer inappropriate ACS discharges annually.
CONTACT INFORMATION
Sonia Singh, MD, MHSc (sonia.singh@fraserhealth.ca), Clinical Assistant Professor, Department of Family Practice, University of British Columbia, Suite 320-5950 University Boulevard, Vancouver, British Columbia V6T 1Z3, Canada; Emergency Physician, Emergency Medicine, Peace Arch Hospital, 15521 Russell Avenue, White Rock, British Columbia V4B 2R4, Canada
Bohdan Nosyk, MA (bnosyk@mail.cheos.ubc.ca), Health Economist, Huiying Sun, PhD (hsun@hivnet.ubc.ca), Statistician, Centre for Health Evaluation and Outcome Sciences, St. Paul's Hospital, 620B-1081 Burrard Street, Vancouver, British Columbia V6Z 1Y6, Canada
James Malcolm Christenson, MD (jim.christenson@gov.bc.ca), Assistant Professor, Department of Surgery, University of British Columbia, 2329 West Mall, Vancouver, British Columbia V6T 1Z4; Emergency Physician, Department of Emergency Medicine, St. Paul's Hospital, 1081 Burrard Street, Vancouver, British Columbia V6Z 1Y6, Canada
Grant Innes MD (ginnes2@providencehealth.bc.ca), Clinical Professor, Department of Surgery, University of British Columbia, 2329 West Mall, Vancouver, British Columbia V6T 1Z4; Chair, Department of Emergency Medicine, St. Paul's Hospital, 1081 Burrard Street, Vancouver, British Columbia V6Z 1Y6, Canada
Aslam Hayat Anis, PhD (aslam.anis@ubc.ca), Professor, Department of Health Care and Epidemiology, University of British Columbia, James Mather Building, 5804 Fairview Avenue, Vancouver, Bristish Columbia V6T 1Z3, Canada; Health Economist, Centre for Health Evaluation and Outcome Sciences, St. Paul's Hospital, 620B-1081 Burrard Street, Vancouver, British Columbia, V6Z 1Y6, Canada
This was an unfunded study. We acknowledge Dianne Calbick for her administrative assistance.
