In England, the National Institute for Health and Care Excellence (NICE) performs appraisals of new medicines in terms of their cost-effectiveness for the National Health Service (NHS), and makes recommendations for the NHS based on these appraisals. Through examination of evidence submitted by medicine manufacturers and evidence review groups, the independent advisory committee, which was called the technology appraisal committee (henceforth referred to as “the TAC”) determines and publishes technology appraisal (TA) guidance, which represents the TAC’s final recommendation regarding the technology in question (1).
The cost-effectiveness of an appraised medicine is typically expressed in terms of cost per healthy year gained, or cost per quality-adjusted life year (QALY) gained, when compared to a comparator medicine. QALYs are calculated by estimating the number of years of life a patient has left after receiving the treatment in question, and weighting each year using a health-related quality-of-life (HRQOL) score (Reference Mehrez and Gafni2). According to the NICE’s guide concerning the TA method (henceforth referred to as “the NICE method guide”), measurement of changes in HRQOL should be based on direct self-reports from patients, and the utility of these changes should be determined by comparing the reported HRQOL with public preferences using a choice-based method. NICE encourages the use of the EuroQol 5 Dimensions (EQ-5D), which has been evaluated in relevant clinical trials for measuring the HRQOL. When EQ-5D data are not available, NICE suggests that utility values be estimated by mapping other HRQOL measures or health-related benefits observed in relevant clinical trials to the EQ-5D, or that EQ-5D data be obtained from existing literature (3).
Some studies have suggested that the EQ-5D lacks sensitivity to changes in health (Reference Devlin and Brooks4;Reference Christiansen, Møller and Kronborg5); therefore, it seems logical that manufacturers, who naturally seek to underline the HRQOL-improving effect of their test medicines, would prefer to use more specialized scales in their clinical trials. In fact, it is much more common for manufacturers to employ disease-specific measures for the health condition of interest than use a general scale. For example, cancer-specific measures that are directly relevant and sensitive to cancer symptoms are used to evaluate the HRQOL of patients with cancer. Two common cancer-specific HRQOL measures are the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core 30 (EORTC QLQ-C30) and the Functional Assessment of Cancer Therapy General (FACT-G) (Reference Aaronson, Ahmedzai and Bergman6;Reference Cella, Tulsky and Gray7).
A descriptive study has examined the HRQOL measures used in clinical trials of health technologies relating to the treatment of breast cancer and pointed out a marked heterogeneity in terms of which measures were used (Reference Reed, Kössler and Hawthorn8). Other studies have focused on the sources of utility values used in manufacturers’ cost-utility analyses in NICE TAs and elucidated lacking or poor compliance with the NICE method guide for HRQOL (Reference Rose, Rice and Craig9–Reference Tosh, Longworth and George11). However, to the best of our knowledge, no previous study has investigated whether the TAC accepted manufacturer-proposed utility values and the reasons for non-acceptance.
The aim of the present study was to investigate whether the TAC’s acceptability of manufacturer-proposed utility values is dependent on the information sources; this was examined by focusing on single technology appraisals (STAs) of cancer medicines, which represent over half of all NICE TAs conducted in the past 5 years.
Methods
Data Sources and Eligibility Criteria
We examined documents that are publicly available from the NICE’s website (www.nice.org.uk). The files in question included the manufacturers’ evidence submission as part of their initial appraisal consultation documents and the TA guidance prepared by NICE. Our inclusion criterion was any STA for a cancer medicine completed between January 2011 and December 2020 because, in cases of multiple TAs, manufacturers’ evidence submission often did not include the information sources of manufacturer-proposed utility values. We excluded appraisals if they were: (i) terminated before completion, (ii) appraisals of medical devices, (iii) appraisals that reviewed previous appraisals, or (iv) appraisals that had been replaced by subsequent reviews. We also excluded appraisals for which the economic model considered health states other than pre-progression, post-progression, and death during cancer treatments because manufacturers need to estimate more utility values when their economic model was more complicated. For example, in the appraisal of trastuzumab emtansine for adjuvant treatment of human epidermal growth factor receptor 2-positive early breast cancer, the manufacturer developed a Markov model with seven health states and estimated six utility values (12). Estimating more utility values may elevate the risk of non-acceptance by the TAC independent of the information sources of manufacturer-proposed utility values.
Data Extraction from Manufacturers’ Evidence Submissions
For each appraisal, we collected data regarding the HRQOL measures from the manufacturers’ submitted evidence. First, we examined the “clinical effectiveness evidence” section of the manufacturers’ evidence submission and identified the clinical trials from which evidence was used in the economic model (such clinical trials are henceforth referred to as “main trials”). If two or more clinical trials were listed, we selected the one from which data were used to estimate effectiveness of the appraised medicine as the main trial.
Second, again examining the “clinical effectiveness evidence” section, we identified the HRQOL measures applied in the main trials. HRQOL measures were classified into four categories: (i) EQ-5D, (ii) EORTC QLQ, (iii) FACT, and (iv) others, respectively. The EORTC QLQ and FACT categories included both general measures (such as the FACT-G and EORTC QLQ-C30) and cancer-specific measures (such as the FACT-Breast or EORTC QLQ-BR23). Instruments other than the EQ-5D, EORTC QLQ, and FACT were classified as “others.”
Third, we identified the information sources of manufacturer-proposed utility values for pre- and post-progression states. The information sources were classified into three categories: (i) EQ-5D, (ii) mapping other measures to the EQ-5D, and (iii) using existing literature or TA guidance. If the manufacturers’ evidence submission adopted a time-to-death approach to estimate the patients’ utility values, we interpreted this as using the same information source for both pre- and post-progression states.
Data Extraction from TA Guidance
We examined TA guidance to determine whether manufacturer-proposed utility values for pre- and post-progression states were subject to objection by NICE. We considered the values to be unacceptable for the TAC if the TAC’s comments included words such as “inappropriate,” “inadequate,” “unfit,” “irrelevant,” and/or “unacceptable.” In contrast, we considered the values to be acceptable if the comments included antonyms of the above-mentioned words, or if there were no comments regarding the manufacturer-proposed utility values.
In cases where the manufacturer-proposed utility values were not accepted by the TAC, we investigated the reason for non-acceptance based on the description in the TA guidance regarding appropriateness of these values. Reasons were classified into three categories by referring to an existing published taxonomy of errors and threats to the credibility of health economic models (Reference Tappenden and Chilcott13) and based on discussions between the authors: (i) inappropriate value for the UK population (e.g., using a higher utility value than that for the general UK population), (ii) inappropriate data adjustment (e.g., no adjustment for age or gender), and (iii) unreliable data source (e.g., using an extremely limited number of subjects). If more than two reasons were mentioned in the TA guidance, we only considered the most discussed one as the cause for non-acceptance in each appraisal because the less-discussed reasons alone were not necessarily sufficient to cause non-acceptance by the TAC.
Statistical Analysis
We categorized manufacturers’ evidence submissions into those in which the information source for the utility values was the application of the EQ-5D in the main trials and those in which utility values were obtained through the other methods, respectively, and then used Fischer’s exact test to assess the hypothesis that there were differences between these groups in the TAC’s acceptance of the utility values. We also compared the reasons for non-acceptance of the manufacturer-proposed utility values stated in the TA guidance from both groups based on the hypothesis that there were differences between them in the reasons for non-acceptance by the TAC. These analyses were conducted separately for pre- and post-progression states. All analyses were performed using StatsDirect ver. 3.3.3 (StatsDirect Ltd., Cheshire, UK). p values less than .05 were considered to be statistically significant.
Results
Characteristics of the Main Trials in Terms of Health-Related QOL Measurement Approach
A total of 414 appraisals were completed between January 2011 and December 2020. Among them, 200 STAs were for cancer medicines. We investigated 136 appraisals, after excluding 64, in the present study. The number of STAs for cancer medicines showed an increasing trend over time; between 2011 and 2014, there were approximately five per year, whereas between 2017 and 2020, there were approximately twenty per year. The number of STAs in which the manufacturers’ evidence submission contained main trials including EQ-5D measurements also increased over time, rising from 10 percent during 2011–2012 to 82 percent during 2019–2020. There were few main trials in which HRQOL was measured only through the EQ-5D; most featured multiple assessments combining both the EQ-5D and cancer-specific measures such as the EORCT QLQ and FACT (Figure 1).
Figure 1. Health-related quality-of-life measurements used in the main trials. Abbreviations: EORTC QLQ, European Organization for Research and Treatment of Cancer Quality of Life Questionnaire; EQ-5D, EuroQol 5 Dimensions; FACT, Functional Assessment of Cancer Therapy; HRQOL, health-related quality-of-life.
Information Sources for Utility Values in the Manufacturers’ Evidence Submissions
The information sources of the manufacturer-proposed utility values for both pre- and post-progression states are shown in Figure 2A,B, respectively. There was an increase over time in the proportion of submissions for which the information source was the EQ-5D. For 2011–2012, the EQ-5D was the information source for 20 percent of the pre-progression state and 10 percent of the post-progression state; however, this rose to 84 percent and 56 percent, respectively, for 2019–2020. When considering the entire research period (i.e., 2011–2020), post-progression utility values, when compared to pre-progression values, were relatively heavily sourced from existing literature or TA guidance rather than the EQ-5D.
Figure 2. Information sources of manufacturer-proposed utility values. Note: (A) Utility values for the pre-progression state and (B) utility values for the post-progression state. Abbreviations: EQ-5D, EuroQol 5 Dimensions; TA, technology appraisal.
Table 1 shows the relationship between HRQOL measures used in the main trials and the information sources of manufacturer-proposed utility values. In ninety-one appraisals, manufacturer-proposed utility values for the pre-progression state were sourced through the application of the EQ-5D in the relevant clinical trials. Eighty-seven of them used the EQ-5D in the main trials, whereas the remaining four used the EQ-5D in relevant clinical trials of appraised medicines other than the main trials. In twelve appraisals, manufacturer-proposed utility values for the pre-progression state were sourced by mapping other HRQOL measures to the EQ-5D using the existing published algorithms. The HRQOL evidence mapped into the EQ-5D were measured in all the main trials. In thirty-three appraisals, manufacturer-proposed utility values for the pre-progression state were sourced from existing literature or TA guidance. Forty-five percent of them measured HRQOL using the EQ-5D or other measures in the main trials, but used existing literature or TA guidance to estimate manufacturer-proposed utility values. For the post-progression state, compared to the pre-progression state, a higher number of appraisals (72 percent) fell into this category.
Table 1. Relationship Between Information Sources for Manufacturer-Proposed Utility Values and Health-Related Quality-of-Life Measurements Performed in the Main Trials
Abbreviations: EQ-5D, EuroQol 5 Dimensions; HRQOL, health-related quality-of-life; TA, technology appraisal.
a These indicate the appraisals where manufacturers used EQ-5D carried out in the relevant clinical trials other than the main trials as the information sources of manufacturer-proposed utility values.
b For the pre-progression state, manufacturers mapped eight EORTC QLQ, two FACT, and two SF-36 to the EQ-5D, and for the post-progression state, they mapped four EORTC QLQ, one FACT, and two SF-36 to the EQ-5D.
The TAC’s Considerations of the Manufacturer-Proposed Utility Values
Table 2 shows the TAC’s judgements on the manufacturer-proposed utility values by the type of utility values (EQ-5D in the main trial or others). Fisher’s exact test revealed no significant differences between the type of utility values in the TAC’s judgement (acceptable or unacceptable) for both pre- and pos-tprogression states. For pre- and post-progression states, 67 percent (58/87) and 56 percent (35/62), respectively, of the manufacturer-proposed utility values derived from the application of the EQ-5D in the main trials were accepted by the TAC; meanwhile, 59 percent (29/49) and 57 percent (42/74), respectively, of the utility values derived from other means were accepted.
Table 2. The TAC’s Judgments on the Manufacturer-Proposed Utility Values
Abbreviations: EQ-5D, EuroQol 5 Dimensions; TAC, technology appraisal committee.
a Fischer’s exact test was conducted to test the hypothesis that there are differences in the proportion of acceptance by the TAC between the “EQ-5D in the main trials” and “others” groups.
In regard to the manufacturer-proposed utility values that were not accepted by the TAC, we compared the reasons for non-acceptance stated in the TA guidance by the type of utility values (EQ-5D in the main trial or others). Table 3 presents information on the statistically significant differences consequently found in the reasons for non-acceptance between the type of utility values for both the pre- and post-progression states. Among the manufacturers’ evidence submissions that featured the EQ-5D in the main trials as a utility-value source, major reasons for non-acceptance were inappropriate values for the UK population (52 percent for pre-progression state and 41 percent for post-progression state) and inappropriate data adjustment (45 percent for pre-progression state and 52 percent for post-progression state). For the other group, reliability of the data source was a common reason for non-acceptance (50 percent for pre-progression state and 34 percent for post-progression state). Cancer type, types of measurement of HRQOL used in the main trials, information sources of manufacturer-proposed utility values, and the decision by the TAC in 136 STAs investigated are shown in Supplementary Table 1.
Table 3. Reasons for the TAC’s Non-acceptance of Manufacturer-Proposed Utility Values
Abbreviations: EQ-5D, EuroQol 5 dimensions; TAC, technology appraisal.
a Fischer’s exact test was conducted to test the hypothesis that there is a difference in the reasons for non-acceptance by the TAC between the “EQ-5D in the main trials” and “others” groups.
Discussion
In the present study, we assessed whether the NICE TAC’s acceptance of manufacturer-proposed utility values is dependent on the manufacturers’ information sources for these values. The number of appraisals for which the EQ-5D was the information source of the manufacturer-proposed utility values increased consistently over the period of 2011–2020. The TAC’s acceptance of the manufacturer-proposed utility values was not dependent on the manufacturers’ information sources, or whether they met the NICE method guide; the primary reasons for non-acceptance by the TAC differed between the manufacturers’ evidence submissions that featured EQ-5D-sourced utility values and those that sourced utility values through other means.
Several previous studies have assessed information sources of manufacturer-proposed utility values (Reference Rose, Rice and Craig9–Reference Tosh, Longworth and George11). These studies highlighted that there is variation in the methods manufacturers use to select and incorporate utility values in economic models, and that a large proportion of manufacturers’ evidence submissions does not include data that accords with the NICE method guide. However, in these studies, little attention was paid to the TAC’s acceptance of the manufacturer-proposed utility values. From the perspective of manufacturers, applying the EQ-5D in main trials seems important for meeting the NICE method guide but, due to the EQ-5D’s lower sensitivity in comparison to disease-specific measures, the EQ-5D is not always useful for elucidating the HRQOL-improving effect of test medicines. Thus, in the present study, we focused on the relationship between the information sources for manufacturer-proposed utility values and the TAC’s acceptance of these values.
The present study showed that, between 2011 and 2012, 20 percent of the manufacturers’ evidence submissions at least partially met the NICE method guide regarding utility values; this percentage is comparable to that reported in a previous study, which showed that between 2004 and 2008, 32 percent of appraisals of cancer medicines satisfied these guides (Reference Tosh, Longworth and George11) and that between 2019 and 2020, approximately 80 percent of submissions were assumed to meet the guides to some extent. Considering the present finding of an upward trend in the proportion of main trials in which the EQ-5D was applied with other measurement tools, it is conceivable that manufacturers are increasingly attempting to propose utility values that meet the NICE method guide.
The present study found that, among the manufacturer-submitted evidence analyzed, utility values for the post-progression state were less likely to meet the NICE method guide than those for the pre-progression state. In other words, a considerable number of manufacturers ceased to use the EQ-5D as a source of utility values when they considered the post-progression state. This may be explained by the limitations concerning investigating the HRQOL of patients after disease progression. Several manufacturers’ submissions mentioned that they collected HRQOL data in the pre-progression state only (14;15). Other submissions ceased using the EQ-5D in the post-progression state because they only collected the EQ-5D at the initial point of progression (16) or because their EQ-5D data were highly immature at the time of the preparation of the evidence submission (17). This indicates that, even in cases when the EQ-5D was applied in the main trials, it was difficult to meet the NICE method guide regarding utility values during the post-progression state. NICE are currently reviewing the method guide to set a hierarchy of preferred methods for measuring HRQOL for when their preferred methods are not available or not appropriate, which will be helpful for manufacturers to estimate utility values during the post-progression state (18).
The present study showed that more than one-third of the appraisals for which manufacturer-proposed utility values were sourced through the application of the EQ-5D in the main trials were not accepted by the TAC. Thus, meeting the NICE method guide is not a sufficient condition for TAC’s acceptance. In contrast, more than half of the appraisals for which manufacturer-proposed utility values were not sourced by the EQ-5D in the main trials were accepted by the TAC if manufacturers considered the best available data. A reason for this may be because manufacturers could refer to several completed appraisals for similar cancer types and treatment lines as over half of all NICE TAs conducted in the past 10 years are for cancer medicines. Utility values based on precedent appraisals would be at a low risk of non-acceptance by the TAC because they have been already discussed within NICE. These two factors may contribute to the finding that the TAC’s acceptance of the manufacturer-proposed utility values was not dependent on whether the utility values were sourced through the application of the EQ-5D in the main trials.
The present study categorized the reasons the TAC did not accept manufacturer-proposed utility values into three groups, and the submissions in question were differentiated depending on whether the information source for the manufacturer-proposed utility values was EQ-5D obtained during the main trials. Issues concerning the reliability of the data source, which could arise as a result of investigation of a small number of subjects or use of an unclear protocol (19;20), are considered resolvable, because manufacturers could design detailed plans for improving their application of EQ-5D measurement in their main clinical trials.
In contrast, issues concerning inappropriate values for the UK population and inappropriate data adjustment are not always resolvable. For example, manufacturer-proposed utility values derived from multiregional clinical trials have the potential for providing inappropriate values for the UK population because there may be differences between the UK and other countries or between patients included in clinical trials and patients in the UK in real-world settings regarding timings of diagnoses, supportive therapies, and intrinsic characteristics (15;21). Several manufacturer-proposed values estimated by mapping other HRQOL measures to the EQ-5D were not accepted by the TAC because they used inappropriate value sets that had not yet been validated. Such cases would be resolved by conducting additional validation studies. Meanwhile, adjustment of utility values based on aging may be difficult because, in many cases, the total evaluation period used for the EQ-5D in clinical trials is shorter than that used in epidemiologic studies referred to in appraisals (22). In short, manufacturer-proposed utility values sourced from applying the EQ-5D in main trials are valuable in terms of showing reliability; however, their use might cause other issues due to the particular characteristics of clinical trials.
In contrast, there were some characteristics of the appraisals that may lead to their acceptance by the TAC. For example, long-term survival follow-up at the time of manufacturer’s submission and frequent EQ-5D measurements during and after treatment discontinuation will result in lowering the risk of non-acceptance. In this case, the manufacturers can confirm the trend of utility decrement over time using evidence obtained through the main trials, and then consider whether they need an adjustment of utility values based on aging (23). In another example, appraisals for first-line cancer treatment or for cancers associated with good prognoses also have the potential to reduce the risk for non-acceptance (24;25). This could be because the general condition of the patients at the time of treatment initiation is more favorable than other cases, which results in a similarity in patients’ general conditions between clinical trial settings and real-world settings in UK. The EQ-5D carried out in these main trials is expected to be considered appropriate for the UK population. These cases were thought to mitigate the unfavorable characteristics of clinical trials.
This study has several limitations. First, we did not consider any medicines or associated indications that were outside the scope of the NICE TAs; therefore, the situation regarding HRQOL evaluations in clinical trials concerning such medicines was not examined in this study. Second, when evaluating the TAC’s acceptance of utility values, we considered only the information source of the manufacturer-proposed utility values, not the absolute values of the HRQOL or the quality of the referenced HRQOL studies. Moreover, we did not consider the quality of the main trials or the cost-effectiveness analyses conducted by the manufacturers; this might affect the TAC’s acceptance as well. Further studies are needed to confirm the impacts of these variables. Third, the presence of subjectivity in the selection of the main trials and the leading cause for non-acceptance cannot be excluded. However, we attempted to base our selection solely on the description in the manufacturers’ evidence submission and the TA guidance to mitigate the subjectivity.
In summary, the present study’s findings suggest that manufacturers make efforts to apply the EQ-5D in their main clinical trials with the aim of utilizing the resultant scores for the NICE TAs; however, to obtain TAC acceptance in this regard, it is not sufficient merely to meet the NICE method guide. Manufacturers must consider in advance the possible differences between their clinical trial settings and real-world settings in UK, as well as the prospective quality of the EQ-5D data available from their trials, and then refine plans for EQ-5D measurement in order to obtain convincing evidence.
Ethical Approval
Ethical approval was not required, as the authors did not collect any personal information, using only aggregate secondary data.
Acknowledgments
We thank Editage for English language editing.
Funding Statement
This research received no specific grant from any funding agency, commercial, or not-for-profit sectors.
Conflicts of Interest
S.T. is an employee of Daiichi-Sankyo Co., Ltd. M.N. does not have any conflict of interest related to this study.
Author Contributions
S.T. was the lead author, and designed the study, gathered the data, conducted the statistical analyses, interpreted the data, and wrote the manuscript. M.N. participated in designing the study, interpreting the data, and reviewing the manuscript.
Data Availability Statement
The datasets generated and/or analyzed during this study are not publicly available, but are available from the corresponding author upon reasonable request.
Supplementary Materials
To view supplementary material for this article, please visit http://doi.org/10.1017/S0266462322000149.
