1. Introduction
Health technology assessment (HTA) supports healthcare resource allocation decisions and is commonly used for drug reimbursement and pricing decision processes. When comparing HTA recommendations across countries, substantial differences are seen. While these are inevitable given the complexity of these processes and the context within which they operate (Banta, Reference Banta2003; Schwarzer and Siebert, Reference Schwarzer and Siebert2009), their magnitude may suggest limitations in the HTA approaches used (e.g. Devlin and Parkin, Reference Devlin and Parkin2004; Dakin et al., Reference Dakin, Devlin and Odeyemi2006; Clement et al., Reference Clement, Harris, Li, Yong, Lee, Manns, Charokopou, Majer, Raad, Broekhuizen, Postma, Heeg, Dakin, Devlin, Feng, Rice, O'Neill and Parkin2009; Mason and Drummond, Reference Mason and Drummond2009; Kanavos et al., Reference Kanavos, Nicod, Pomedli and van den Aardweg2010; Fischer, Reference Fischer2012; Nicod and Kanavos, Reference Nicod and Kanavos2012; Fischer et al., Reference Fischer, Rogowski and Stollenwerk2013; Cerri et al., Reference Cerri, Knapp and Fernandez2014; Dakin et al., Reference Dakin, Devlin and Feng2015; Charokopou et al., Reference Charokopou, Majer and Raad2015). Better understanding the reasons for such disparities may help reinforce these processes. Ultimately, this will lead to fairer access for patients, better system predictability for marketing authorisation holders, and ensure a more efficient use of health care resources.
Two recent studies (reported in six papers) adopted qualitative and quantitative research designs, respectively, to explore the reasons for cross-national differences in HTA recommendations for drugs (Maynou and Cairns, Reference Maynou and Cairns2015, Reference Maynou and Cairns2016, Reference Maynou and Cairns2019; Nicod and Kanavos, Reference Nicod and Kanavos2016a, Reference Nicod and Kanavos2016b; Nicod, Reference Nicod2017). The qualitative study investigated these differences in four countries for 10 orphan drugs. To do so, a methodological framework was developed allowing to inductively identify the factors contributing to decisions by adopting a systematic approach to comparing these processes across drugs and countries (Nicod and Kanavos, Reference Nicod and Kanavos2016a). The main factors contributing to explaining differences across countries included: (a) differences in the evidence used within the decision process, (b) different interpretations of the same evidence, (c) different consideration of decision moderators and (d) differences resulting from the HTA approach used (Nicod, Reference Nicod2017). The social value judgements made during the deliberative phase of these processes were also captured together with how they influenced the decision (Nicod and Kanavos, Reference Nicod and Kanavos2016b).
The quantitative study tested the extent to which a number of micro- and macro-level factors, identified from a review of the literature, contributed to explaining these differences (Maynou and Cairns, Reference Maynou and Cairns2016). The analysis involved ten European countries and 199 drugs, and highlighted trends across all countries. Findings showed that certain organisational factors are significantly associated with lower or higher probabilities of reimbursement. This included, for instance, the external review of evidence or the requirement for an economic evaluation, which were negatively related to the probability of reimbursement. Positive examples were seen in some countries when previously a positive assessment had been issued by the National Institute for Health and Care Excellence (NICE), or when the drug was considered cost-effective by NICE or the Scottish Medicine Consortium (SMC).
The objective of this study was to bring together these two studies and generate a fuller picture as to why countries make different HTA decisions. We adopted a mixed methods approach, which allows us to integrate and interpret results from two studies that used different methodological approaches to answer the same research question. This paper is organised as follows: Section 2 describes the methods used, Section 3 presents the results and Section 4 discusses the findings and conclusions.
2. Methods
1.1 Study design
A parallel convergent mixed methods research design was adopted, where both qualitative and quantitative studies were first conducted concurrently and independently, and then merged and compared in the interpretation phase (eAppendix 1) (Creswell and Plano Clark, Reference Creswell, Plano Clark, Creswell and Plano Clark2011a). Combining these studies will help better understand the HTA decision process, and bring together the strengths and non-overlapping weaknesses from the approaches used (Creswell and Plano Clark, Reference Creswell, Plano Clark, Creswell and Plano Clark2011b). These two studies were directly comparable given they addressed the same research question in the same countries, using different research designs.
A conceptual framework was developed by the authors that integrated and organised the findings from the two studies. This two-dimensional framework distinguishes the factors driving the HTA decision processes identified quantitatively and/or qualitatively and whether they are macro- or micro-level considerations. Macro corresponds to system-level variables (common to across all drugs) and micro to product specific variables (different across case studies). We then explored the interface between (1) macro-level factors identified in the quantitative vs those in the qualitative studies and (2) micro-level factors identified in the quantitative vs those in the qualitative studies. Case study analyses were conducted to explore and illustrate these interactions. We then investigated whether the factors identified in the quantitative and in the qualitative studies were congruent, complementary or discrepant. This assessment was carried out by the authors. Case studies were used to illustrate and confirm our interpretation. A joint display merging these data for better visualisation was created. The comparative method was used (Gerring, Reference Gerring and Gerring2007), which involved several case studies being compared within-cases and across-cases. These case studies form the integration phase of this mixed methods project, where findings from both qualitative and quantitative analyses are being compared, contrasted and built upon (Mertens, Reference Mertens2011). Each case corresponds to the drug and indication pair under consideration, and the phenomena being further explained is the HTA recommendation issued in each country.
2.2 Sampling
The HTA bodies common to the two original studies were selected. Selection criteria included whether their advice relies on the assessment of clinical and/or cost-effectiveness, whether their recommendations are publicly available, and whether they are based in Europe. These comprised NICE in England, SMC in Scotland, the Haute Autorité de Santé (HAS) in France and the Dental and Pharmaceutical Benefits Agency (TLV) in Sweden. Their HTA recommendations rely on the assessment of clinical and cost-effectiveness assessments (NICE, 2015; SMC, 2016; TLV, 2016), and are in the form of ‘list’, ‘restrict’ or ‘reject’ the drug for reimbursement. It is somewhat different for HAS, which focuses on the assessment of clinical benefit (‘Service Médical Rendu’, SMR) and improvement in clinical benefit (‘Amélioration du Service Médical Rendu’, ASMR). The SMR drives the coverage rate (e.g. 0%, 15%, 35% and 65%), while the ASMR the pricing scheme (e.g. internal reference pricing or price negotiation within European price levels) (HAS, 2014). Drugs considered irreplaceable and expensive by the Ministry of Health are reimbursed at 100% (Chevreil et al., Reference Chevreil, Berg Brigham, Durand-Zaleski and Hernandez-Quevedo2015).
The main variable of interest was the HTA recommendation issued in each country. The recommendation was considered restricted when the indication approved differed from the indication in the reimbursement application.
Purposive sampling was used to select the four case studies of drug-indication pairs, if possible, that were analysed in both studies (Maynou and Cairns, Reference Maynou and Cairns2016; Nicod and Kanavos, Reference Nicod and Kanavos2016a, Reference Nicod and Kanavos2016b; Nicod, Reference Nicod2017). The following selection criteria are used:
(1) One compound per therapeutic area (oncology, musculoskeletal diseases and central nervous system diseases) and one orphan drug;
(2) Having received non-homogenous HTA recommendations (when possible);
(3) Illustrating the reasons identified for cross-country differences.
A number of drug and indication pairs fulfilled the first two selection criteria. The objective of these case studies was to illustrate the reasons identified for cross-country differences across the two studies, characterised by the conceptual framework developed. On this basis, criteria #3 was used to select the four case studies based on whether they are representative of the different reasons for cross-country differences.
This resulted in a selection of four drug and indication pairs (Table 1). Building on the methodology developed in the qualitative study, the non-orphan 3 drugs not included in the qualitative study were systematically analysed such that they are all comparable. Three of these drug-indications received diverging HTA recommendations across countries. One was positively appraised when assessing cost-effectiveness, and rejected when assessing clinical benefit by HAS (mifamurtide), while the other two received positive assessments of clinical benefit and improvement in clinical benefit by HAS, but received contradictory assessments (e.g. list, restrict or reject) on cost-effectiveness grounds in the other countries.
Table 1. Case studies, HTA recommendations and micro-level variables (context- and disease-specific)
SMR, Service Médical Rendu; ASMR, Amélioration du Service Médical Rendu; RP, reference pricing; EU, price negotiated within European price levels; BSC, best supportive care; TLV, Dental and Pharmaceutical Benefits Board; HAS, Haute Autorité de Santé; SMC, Scottish Medicines Consortium; NICE, National Institute of Health and Care Excellence; HTA, health technology assessment.
a The outcome of the assessment for tocilizumab depended on its place in the therapeutic strategy: DMARD = after DMARDs as an alternative to TNF-alpha inhibitors; TNF = after TNF-alpha inhibitors, as an alternative to rituximab.
b Restriction: two or more disabling relapses in one year and with one or more gadolinium-enhancing lesions on brain magnetic resonance imaging (MRI) or a significant increase in T2 lesion load compared with a previous MRI.
c The ICER depends on the place of tocilizumab in the sequence of treatments.
Indications:
• Mifamurtide: In combination with post-operative multi-agent chemotherapy for the treatment of high-grade resectable non-metastatic osteosarcoma after macroscopically complete surgical resection, in children, adolescents and young adults.
• Gefitinib: For the treatment of adult patients with locally advanced or metastatic non-small cell lung cancer with activating mutations of epidermal growth factor receptor tyrosine kinase (EGFRTK).
• Natalizumab: As single disease modifying therapy in highly active relapsing remitting multiple sclerosis (RRMS) for the following patient groups; patients with high-disease activity despite treatment with beta-interferon and patients with rapidly evolving severe RRMS.
• Tocilizumab: In combination with methotrexate, for the treatment of moderate to severe active rheumatoid arthritis in adult patients who have either responded inadequately to, or who were intolerant to, previous therapy with one or more disease modifying anti-rheumatic drugs or tumour necrosis factor antagonists.
3. Results
3.1 Conceptual framework
Figure 1 illustrates the conceptual framework developed distinguishing macro- and micro-level factors, and those identified within the quantitative and/or qualitative analysis. At the macro-level, factors were clustered as organisational, process-specific and methodological variables (Table 2). Some factors were excluded from the quantitative econometric model due to too little variation or correlation with other variables, but were nevertheless considered in this study as they may have played a role in a case-by-case basis (qualitatively). None of the macro-level factors were explicitly captured in the qualitative study, apart from on methodological variable as to whether an economic model is being considered.
Figure 1. Conceptual framework describing the variables driving HTA decisions identified qualitatively and quantitatively. The figure illustrates the conceptual framework integrating the variables identified qualitatively and/or quantitatively as potentially explaining cross-country differences in the HTA recommendations issued, and whether they are micro-level or macro-level variables. The framework also highlights those variables that were statistically significant in explaining differences across countries (positively or negatively). Legend: 1considered for the quantitative assessment, but not included in the econometric model as insufficient variation; 2not captured in the qualitative assessment; 3not included in the quantitative assessment; in bold: statistically significant effect on probability of reimbursement; NICE: National Institute for Health and Care Excellence; SMC: Scottish Medicines Consortium, TLV: Dental and Pharmaceutical Benefits Board; HAS: Haute Autorité de Santé; ICER: incremental cost-effectiveness ratio.
Table 2. Macro-level variables identified in the quantitative research study
Source: summary of factors driving decisions identified in Maynou and Cairns (Reference Maynou and Cairns2015, Reference Maynou and Cairns2016).
a In France, economic evaluations were not required before 2013.
b Reference pricing, if the drug-indication is adding value ASMR I, II, III. Negotiation, if is not adding value ASMR IV, V.
At the micro-level, three groups of considerations were identified both within the quantitative and qualitative studies: (a) context-specific factors, (b) disease-specific factors and (c) those related to the decision process (Table 1, eAppendix 2). Context-specific factors were defined as those pertaining to one specific agency. These included time-dependent factors corresponding to the date when the HTA recommendation was issued in each country, so as to assess the influence the first decisions had on the consecutive ones (Maynou and Cairns, Reference Maynou and Cairns2016). They may also be process-specific (e.g. ability to implement MEAs, ICER estimate, ability to impose restrictions or re-assessments, or lower discount rates), or correspond to elicited societal preferences relating to circumstances where greater uncertainty or ICERs may be accepted (e.g. disease severity in Sweden, lower discount rate, NICE end-of-life treatment, SMC modifiers) (Maynou and Cairns, Reference Maynou and Cairns2016; Nicod and Kanavos, Reference Nicod and Kanavos2016b; Nicod, Reference Nicod2017). Disease-specific factors relate to the context of the disease the drug is treating (e.g. rare condition, disease stage, availability of alternative treatments). Decision-process variables relate to the evidence used in the decision process and its critical appraisal during the deliberative process (Nicod and Kanavos, Reference Nicod and Kanavos2016a).
3.2 Exploring the interface between qualitative and quantitative results
3.2.1 Quantitative vs qualitative macro-level variables
Organisational variables: The inclusion of an external independent review (e.g. NICE Evidence Review Group, ERG) was associated with a higher probability of rejection in the quantitative study (Maynou and Cairns, Reference Maynou and Cairns2016). In the qualitative analysis, NICE's ERG input was captured as ‘stakeholder input’, and may have positively or negatively influenced the appraisal committee's judgement. For example, gefitinib was considered cost-effective by NICE and TLV, but not by SMC. NICE's positive outcome related to the ERG's correction of the manufacturer's economic analysis in using gefitinib as a baseline instead of paclitaxel plus carboplatin. The other organisational factors were not captured in the econometric model nor in the qualitative study.
Process-specific variables: The initiator of the HTA process was a variable included only in the quantitative analysis. In most of the cases the manufacturer applies for reimbursement (e.g. Belgium, France, Poland, Portugal and Sweden), however, in some countries, submission is requested by the Department of Health (e.g. England, Spain), from the body in charge of HTA (e.g. Netherlands) or as an automatic procedure (e.g. Scotland, Germany). In these last cases, the manufacturer is asked to submit an application. Results showed that the manufacturer's initiation of the HTA process was associated with a higher probability of rejection.
The impact of transparency or of an appeal process was not captured in the econometric model nor qualitatively. It may, however, have influenced the level of detail captured during the qualitative analysis, with fewer explanatory variables captured for the lesser transparent bodies. Lower levels of stakeholder involvement were associated with a higher probability of acceptance in the quantitative study. This was not observed in the qualitative results, where stakeholder involvement contributed to reducing uncertainty around costs or effects, thus influencing the decisions both positively and negatively.
Methods and HTA approaches: A requirement for an economic evaluation or budget impact was quantitatively associated with a higher probability of rejection (Maynou and Cairns, Reference Maynou and Cairns2016), but was not captured explicitly qualitatively. HAS does not require an economic evaluation, whereas the HTA recommendation is about the coverage rate and applicable pricing scheme. It would, therefore, be linked to the pricing decision also captured in the model. Unless the evidence presented was insufficient, all drugs would receive a positive reimbursement recommendation, where the price would be managed separately.
3.2.2 Quantitative vs qualitative micro-level variables
Disease-specific variables: The disease-specific variables were excluded from the quantitative analysis (Maynou and Cairns, Reference Maynou and Cairns2016), but captured in the qualitative analysis (Figure 1). ‘Orphan status’ was accounted for explicitly or implicitly (Nicod and Kanavos, Reference Nicod and Kanavos2016b). The former relates to greater acceptance of uncertainty in the economic analysis and of a higher cost/QALY due to its orphan status, e.g. within SMC's modifiers. The latter refers to cases where the uncertain nature of the evidence produced due to a condition's rarity was recognised during deliberations. One example of this was mifamurtide, where the difficulties in recruiting sufficient patient numbers and conducting subgroup analyses were recognised by NICE, and greater uncertainty in the evidence presented was deemed acceptable. ‘Disease stage’ was not explicitly discussed and reported (Nicod, Reference Nicod2017), but was accounted for indirectly when considering results from different subgroups of patients or lines of treatment, for which the ICER was more acceptable. A case in point is gefitinib, where, based on the grounds of cost-effectiveness, NICE imposed a restriction to patients receiving first line treatment. NICE recognised that patients receiving second-line treatment may have more severe disease and slightly worse utility than patients receiving first-line treatment. The ICER for second-line use was still too high (£40,048/QALY compared to gemcitabine and carboplatin).
‘Availability of an alternative treatment’ was captured in the qualitative study when different aspects of unmet need were discussed. In this respect, the lack of treatment alternatives, the impact on the patient without the treatment or the need for additional treatment alternatives was recognised in the different contexts (Nicod and Kanavos, Reference Nicod and Kanavos2016b). For example, for natalizumab, the need for improved care pathways for multiple sclerosis, or for mifamurtide, the need for new options improving overall survival, were recognised by some appraisal committees. In this latter case, HAS merely highlighted that alternative treatments exist.
Context-specific variables: In the quantitative analysis, the time dependent variable (e.g. influence of first HTA recommendation on others) was not significant across all drugs and countries, while there was a higher probability of reimbursement in other countries after a positive recommendation by NICE (Maynou and Cairns, Reference Maynou and Cairns2016). Even if there is limited literature looking at the effects of NICE recommendations on other HTA bodies, this finding is in line with recent research. Wijnands et al. (Reference Wijnands, Kreeftmeijer, Kooreman, Wagner and Van Engen2016) found that many agencies outside the UK adopt similar methodologies to NICE and that 60% of their sample echoed NICE recommendations. In the quantitative analysis, this variable was introduced as a proxy for the quality of the supporting evidence, given that NICE HTA analyses are considered among the most complete and strict.
Upon further examining the four case studies, the time elapsed between the first and last assessment made across all four countries was found to be less than 9 months for two of the drugs and up to 2.5 years for mifamurtide and tocilizumab. Despite these time differences, the same primary trials were considered and the differences between decisions could not be explained by any additional evidence having been generated during that period. In terms of the sequence of the recommendations, no clear pattern was observed apart from HAS tending to be first (mifamurtide and tocilizumab). The same applies to the quantitative examination of the effect of the economic crisis period (2008 onwards), which was negatively related to the probability of reimbursement (Maynou and Cairns, Reference Maynou and Cairns2015, Reference Maynou and Cairns2016). This was not captured qualitatively.
A cost-effective drug for NICE and SMC was likely to receive a positive HTA recommendation in the quantitative study (Maynou and Cairns, Reference Maynou and Cairns2016). The cost-effectiveness variable was created as a binary variable using the NICE threshold (i.e. ICER above or below £30,000 per QALY). This was in line with the qualitative findings (Nicod, Reference Nicod2017), which additionally captured how these ICERs may have been moderated to acceptable levels. For example, NICE's revised ICER for mifamurtide ranged between £56,667 and £60,205/QALY after including a patient access scheme. Additionally, the Committee accepted a lower discount rate (1.5%) on effects as these were considered to be substantial in restoring health and sustained over a long period. This, together with the treatment's innovativeness and recognition that the positive effect on quality of life was likely not captured by the evidence appraised, resulted in an acceptable ICER (£36,000/QALY). A lower discount rate was also accepted by SMC for similar reasons (rendering the ICER more acceptable), whereas the high ICER was accepted by TLV due to the condition's severity. The main contrast was seen in France, where the evidence presented demonstrated only a weak benefit (due to the poor design and conduct of the trial) and, as a result, it was rejected for reimbursement.
These moderator factors relate to context-specific considerations, which include different acceptance levels or willingness-to-pay thresholds for an ICER, the ability to implement MEAs, accept greater uncertainty or ICERs under special circumstances (e.g. NICE's end-of-life criteria). These are key determinants in explaining differences across countries, are often used in combination, and have shown to contribute to rendering an ICER ‘cost-effective’.
Some of these moderating factors were captured quantitatively, but measured independently from the ICER. Results show a positive association between the HTA recommendation and the inclusion of an MEA or the fulfilment of NICE's end-of-life criteria (at a 90% confidence interval) (Maynou and Cairns, Reference Maynou and Cairns2016). This was illustrated in the case of gefitinib, which was positively appraised by NICE (given it met NICE's end-of-life criteria) after agreeing with the manufacturer on a fixed price under the patient access scheme reducing the ICER to an acceptable level (£23,600/QALY gained for a maximum of six cycles (mean 5.4 cycles) and £64,500 per QALY gained for a maximum of five cycles (mean 4.6 cycles)). However, the application of these moderating factors varied depending on the setting, as seen in Scotland, where the case presented was not sufficiently robust to demonstrate its cost-effectiveness. In Sweden, different arguments were put forward where, despite being more expensive than its comparators, gefitinib received a positive recommendation thanks to its high level of clinical benefit combined with a significant increase in quality of life.
Decision process: Another important component driving HTA decisions relates to what goes on during the deliberative process. The acceptability of uncertainty or high ICERs depends on how uncertainty is perceived and managed. The qualitative study was able to capture this and found a high level of heterogeneity throughout the decision process (Figure 1) (Nicod and Kanavos, Reference Nicod and Kanavos2016b; Nicod, Reference Nicod2017).
First, differences were identified in the evidence appraised (Nicod, Reference Nicod2017). In the case of natalizumab, for instance, all considered a phase III placebo-controlled study (AFFIRM), except TLV who considered an indirect comparison of natalizumab with another disease-modifying therapy as primary evidence. Another example includes the economic models considered for tocilizumab when compared to TNF-alpha inhibitors, where NICE and SMC considered cost-utility models and TLV an unspecified model (likely a cost-minimisation model). Similar treatment sequences were included by NICE and SMC. Tocilizumab was considered dominant by TLV (similar effect and equivalent price), cost-effective by SMC, but restricted by NICE because of the interpretation of the treatment sequences presented (discussed in the next sub-section). These differences may be a consequence of the type of evidence required or preferred by the agencies (Nicod, Reference Nicod2017; Nicod et al., Reference Nicod, Berg Brigham and Durand-Zaleski2017), or by other contextual considerations, such as local clinical pathways or guidelines. Quantitatively, the former would be related to the HTA ‘Methods’ variable, but were not captured in the econometric model being explored.
Second, differences were seen in how the same evidence appraised across countries was interpreted. In total, 46 concerns were raised by the four agencies across the four drugs, of which only 45% (21/46) were common to at least two agencies (eAppendix 2). In our previous example, the same economic models considered for tocilizumab were interpreted differently by NICE and SMC. In the sequence of treatments considered by SMC, probabilistic sensitivity analysis resulted in a high probability of being cost-effective. By contrast, NICE considered different treatment sequences following the ERG's review, which had implications on costs and led to imposing a restriction in the indications. Such subtleties relate to the decision-makers' scientific value judgements, the differing scope and remit of the appraisal committees in interrogating the manufacturers' submissions, or the approaches and processes in place to deal with uncertainty in a particular context. These were not captured in the quantitative analysis.
Differences were also seen in the way the same issue raised by several agencies was dealt with. Across the four study drugs, 48% (10/21) of concerns relating to the clinical evidence and common to at least two agencies were managed differently (eAppendix 2). For example, the ethnic origin of the gefitinib trial population was considered different from the local context by all agencies, but considered acceptable for NICE based on clinical expertise and for TLV based on the high benefit rendered in a subgroup of patients. While such a level of detail may not be captured quantitatively, the HTA methods or processes may explain how uncertainty is managed within the different contexts. In our example, stakeholder involvement has shown to help deal with uncertainty.
A number of qualitative criteria played an important role during deliberations. They include more common considerations, such as unmet need or disease severity, or other more distinct circumstances of a particular disease. For example, a number of these considerations were accounted by NICE for mifamurtide and provided by patient experts: impact on patients, their families and friends of diagnosing and treating osteosarcoma, the curative intent of the treatment, or high unmet need. The chronicity and severity of the disease were also considered relevant for the final decision on natalizumab by NICE and HAS, whereas this had no influence at all for SMC and TLV. The administration benefit and availability of few alternatives were also valued by all agencies for gefitinib and natalizumab. Most of these relate to disease-specific variables, some of which were provided through the stakeholder input and, apart from the orphan status, availability of alternatives or disease stage (which we have seen may be captured in various ways), their nuances were not captured quantitatively.
3.3 Congruency, complementarity or discrepancy of findings
Based on the analysis above of the relationships between the factors driving HTA decisions identified quantitatively and qualitatively, we examined whether the 25 factors examined were congruent, complementary or discrepant (Figure 2). Only two of the factors common to both studies were congruent: the inclusion of MEAs and NICE end-of-life treatments associated with higher probabilities of positive reimbursement. Qualitative findings additionally showed that the application of MEAs and NICE's end-of-life was not always consistent, where it may have been implemented in one country and not another, and may also depend on the Committee involved (Nicod et al., Reference Nicod, Berg Brigham and Durand-Zaleski2017).
Figure 2. Linking the variables identified quantitatively and quantitatively as explaining cross-country differences in HTA recommendations. Congruent, complementary and discrepant examples. The figure summarises how those variables identified both quantitatively and qualitatively as explaining cross-country differences relate to each other, and whether the evidence provided was congruent, complementary or discrepant. Macro-level variables were differentiated from micro-level variables. Legend: √: whether the variable was identified quantitatively or qualitatively; ‘-’: the variable was significantly associated with a negative probability of reimbursement; ‘+’: the variable was significantly associated with a positive probability of reimbursement; ICER: incremental cost-effectiveness ratio; NICE: National Institute for Health and Care Excellence; SMC: Scottish Medicines Consortium, TLV: Dental and Pharmaceutical Benefits Board; HAS: Haute Autorité de Santé.
One factor, review of evidence, was both congruent and discrepant. Results from the quantitative study showed that the inclusion of an external review of the evidence was associated with negative recommendations. Qualitatively, the external review of the evidence did have an influence on the interpretation of the evidence, contributing to cross-country differences; however, the influence was not only negative as it may also have contributed to a positive recommendation.
In contrast, two other factors gave discrepant conclusions: stakeholder involvement and external review of the evidence. These were associated with a higher probability of rejection in the quantitative analysis, but showed different contradictory trends in the qualitative results (either positive and/or negative influence on the decisions). All other factors identified within one or both studies were complementary, explaining and enhancing each other.
4. Discussion and conclusion
This study illustrates the added value of using mixed methods to further explore the reasons for different HTA recommendations across countries. The methodological approach developed and applied is novel. Qualitative findings helped explain quantitative ones, while quantitative findings enhanced qualitative ones. The first step, creating the conceptual framework, contributed to merging and organising findings from the two studies. The second step, examining and illustrating the relationships between variables through case studies, allowed for a better understanding of the ways in which these factors affect the appraisal processes. On this basis, we were then able to define the relationships between the variables and what we can learn from them. In this section, we discuss how the results have advanced the understanding of HTA decisions across countries and how they fit with the existing literature.
4.1 Factors driving HTA decisions – what have we learnt
The variables identified as being congruent (inclusion of MEAs or patient access schemes, or NICE end-of-life treatments) do not add any additional insights beyond findings from the individual studies. These were identified in the qualitative study as decision moderators allowing countries to modulate the ICER to acceptable levels (Nicod, Reference Nicod2017), and in the quantitative study through the positive association between reimbursement decision and their inclusion. This is not surprising given that the former allows the payer to share the financial risk with the manufacturer (Ferrario and Kanavos, Reference Ferrario and Kanavos2015), while the latter concedes a higher willingness-to-pay.
The complementary variables provide additional insights and can be distinguished in terms of: (1) those captured both qualitatively and quantitatively providing similar or different (complementary) insights and (2) those captured either qualitatively or quantitatively.
The first type includes the ‘cost-effective ICER’, which was quantitatively more likely to receive a positive recommendation. Qualitatively, a ‘cost-effective ICER’ was often a moderated one (through a combination of mechanisms, e.g. inclusion of an MEA or PAS), thus allowing the ICER to be decreased or uncertainty to be rendered more acceptable (Nicod, Reference Nicod2017). Additionally, when including only six countries in the econometric model, the incidence rate had a negative effect on the probability of reimbursement because, other things equal, it is associated with a larger budget impact (Maynou and Cairns, Reference Maynou and Cairns2019). This was also illustrated in the qualitative results, where those drugs used to treat fewer than 20 patients per year were more likely to receive a negative reimbursement recommendation than the ‘more prevalent’ rare conditions had in terms of the HTA recommendations issued (Nicod, Reference Nicod2017).
Second, many variables were identified only qualitatively: ‘orphan status’, ‘availability of other treatments’, ‘disease stage’, ‘lower discount rate’, imposing a re-assessment or restriction, and elicited societal preferences. These were, however, excluded from the econometric model. Another example is the influence disease severity has in Sweden, where a higher willingness-to-pay is accepted compared to the lesser severe diseases (Svensson et al., Reference Svensson, Nilsson and Arnberg2015). Qualitative results show that these variables play an important role in the decisions, and should therefore be considered together with the quantitative results. The variables captured only quantitatively included ‘initiator of the process’, ‘budget impact’, ‘economic crisis’, or ‘positive decision by NICE’. These would not be captured qualitatively, as this would require a large sample size and they are independent and do not necessarily require more depth to understand their meaning.
Results also identified two variables that gave contradictory results (stakeholder input and external review of the evidence processes). These emphasises the need for caution when interpreting results from individual studies and the benefit of using mixed methods approaches. The stakeholder input often contributed to confirming (on not) the plausibility of an uncertain claim, most often rendering uncertainty acceptable. When analysing the quantitative results in more depth, which suggested that lower levels of stakeholder involvement were associated with a higher probability of acceptance, these were likely driven by the two countries with contrasting levels of stakeholder involvement and trends in the HTA recommendations issued (e.g. low in Spain and high in England). This is an example of where the analyst is vulnerable to being misled by a quantitative approach. However, by including only six countries in the quantitative model, it was no longer significant (Maynou and Cairns, Reference Maynou and Cairns2019).
Only two variables were congruent, further emphasising the added value of combining these approaches given that they do not necessarily capture the same variables. Results also allow the limitations of using one approach or another to be better understood. Quantitatively, some relevant variables were excluded or not initially included in the model. Identifying the variables of interest inductively, as was done in the qualitative strand, would minimise this. The variables included in the quantitative strand were assumed to be independent, while qualitative results showed that one variable may be due to a combination of events. The depth of what occurred during the deliberative processes was not captured quantitatively, further emphasising the importance of combining different research approaches. Qualitatively, it was not possible to capture the effect of certain variables requiring a large sample size. These are usually independent criteria, which do not necessarily require greater depth to understand the phenomena.
This study allows a better understanding of why countries make different reimbursement recommendations. The implications from having differences in HTA recommendations between countries may imply a failure to secure value for money and differences in patient access to medicines (and vice versa). The HTA recommendation does not always match the reimbursement status (Garau and Mestre-Ferrandiz, Reference Garau and Mestre-Ferrandiz2009); however, it is often a good indicator. Further research could usefully explore the extent to which patient access to treatments is correlated with HTA recommendations.
While cross-country differences in reimbursement recommendations are legitimate as they reflect local decision-making (e.g. local clinical practice, preferences, priorities), they may also be a consequence of differences in the methodologies, timelines, evidence considered and review of evidence across settings and it is important to identify and understand these. Whether these differences are justified is not for us to say, but may be questionable when they result from differences in methodologies, timelines, evidence and review of the evidence. The current EC proposal aiming to harmonise HTA processes across EU Member States through joint clinical assessments could be a good way forward to minimising these cross-country differences.
A first limitation is that our results are not generalisable. The case study approach used aims to explore the relationships between two studies that addressed the same research question using different research designs. This is yet an unknown and complex area, and a more in-depth understanding of what is happening in each setting, at different levels, and how this may contribute to explaining cross-national differences, is needed before making generalisations. However, these results contribute to gaining a fuller understanding of what may be driving differences across countries and provide illustrative examples of how the factors used to reach such decisions may be being interpreted. Second, given that this analysis focuses on a set of criteria and four case studies, there may also be unsubstantiated or missing observations which were not captured by the authors during the analyses.
4.2 How these findings fit with existing knowledge
Both quantitative and qualitative results suggest that a ‘cost-effective’ ICER (for NICE and SMC) is a predictive factor for positive recommendations. This is in line with previous research (Devlin and Parkin, Reference Devlin and Parkin2004; Cerri et al., Reference Cerri, Knapp and Fernandez2014, Reference Cerri, Knapp and Fernandez2015; Dakin et al., Reference Dakin, Devlin and Feng2015; Svensson et al., Reference Svensson, Nilsson and Arnberg2015), which also showed that the probability of rejection increases with the cost/QALY estimate. When exploring reasons for differences across several countries using multivariate analyses, one study recognised that the ICER alone is not sufficient to predict HTA recommendations (Cerri et al., Reference Cerri, Knapp and Fernandez2014). This is because a decision often relies on the decision-maker's judgement of uncertainty in the evidence appraised and what happens during the deliberative process. Indeed, only 30% of the variability across countries (England, Scotland, the Netherlands and France) could be explained by their multivariate model developed, possibly because of a range of factors that are highly specific to each agency (Cerri et al., Reference Cerri, Knapp and Fernandez2015). A similar scenario was reflected in our quantitative model, where many variables were excluded because of comparability issues across countries.
Some of these variables were identified within the qualitative strand and this study further contributes to understanding how they relate to macro- or micro-level factors. These factors included process-specific variables, societal preferences, as well as the interpretation of the evidence. Indeed, while the (scientific and social) value judgements made by the committee members played an important role in the decision-making, their judgements and the combinations of factors rendering the ICER acceptable may not have been consistently applied across the cases.
Qualitative and quantitative results (individually or combined) corroborate existing evidence on factors influencing HTA recommendations and, additionally, improve the understanding of how they play that role. For example, Dakin and colleagues identified a number of factors influencing decisions that were also identified in our study. These included no alternative treatments, patient group submissions, end-of-life criteria, disease severity, or the lack of effect on the date of the decision on NICE decisions (Dakin et al., Reference Dakin, Devlin and Feng2015). Our results further contribute to understanding their application in practice. For example, the availability of an alternative treatment was excluded from the econometric model but captured differently qualitatively. It was a proxy for unmet need, which relied on additional qualitative criteria such as disease severity, effects or side effects of current treatment alternatives, requiring an in-depth understanding of local clinical pathways and patient experiences. An earlier study by the same authors found that patient submissions were more likely to increase the probability of endorsement for routine care rather than for restricted use (Dakin et al., Reference Dakin, Devlin and Odeyemi2006), echoing our qualitative findings and contradicting our quantitative ones (where it had a negative effect on the probability of reimbursement).
In conclusion, this mixed methods study contributes to generating a fuller picture of why different countries make different HTA recommendations. The methodological contribution is strong given the unique approach used in comparing two separate studies that explored the same research question from two different angles. Combining both approaches enhanced the findings and generated a better understanding of how the variables are reflected in practice, as illustrated by the case studies, further contributing to the validation of the respective results. We conclude that, although these processes are systematic and rely on evidence-based medicine, a component of these decisions relies on judgement made during the deliberative process, and are often a consequence of a combination of factors.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S1744133119000239.
Financial support
Financial support for this study was partially provided to all authors by the European Commission's 7th Framework Programme and undertaken under the auspices of Advance-HTA (Grant # 305983). The funding agreement ensured the authors' independence in designing the study, interpreting the data, writing and publishing the report. The results presented here reflect the authors' views and not those of the European Commission. The EC is not liable for any use of the information communicated.
Conflict of interest
EN reports grants from H2020 EC grant, personal fees from Dolon Ltd, personal fees from Consulting/honorary fees, outside the submitted work. LM, EV and JC do not report any conflicts of interest.
