1. Introduction
Health technology assessments (HTAs) are widely used as a policy and administrative tool to appraise the therapeutic value, or cost effectiveness, of new medicines to inform reimbursement decisions (Freeman, Reference Freeman, Marmor, Freeman and Okma2009). Labelled the ‘fourth hurdle’ in the regulation of medicinal products, HTA's policy implications are wider than those of the other three hurdles: quality, safety and efficacy. Denying access to a medicine on the basis of cost effectiveness frequently leads to public discontent, especially amongst the patient groups who are affected. On the other hand, policymakers are concerned about the large sums of money that are involved when expensive, and sometimes only marginally effective, new treatments enter the market. As HTA organisations have been established in different countries, it has become clear that the outcomes of medicine appraisals vary (e.g. Nicod and Kanavos, Reference Nicod and Kanavos2012; Nicod, Reference Nicod2017), even though the same scientific evidence in the form of randomised controlled trials (RCTs) is available. Explanations for variations include different administrative rules such as information requirements (Kanavos et al., Reference Kanavos, Nicod, van den Aardweg and Pomedli2010), factors relating to how HTA bodies are designed and the characteristics of the drug reimbursement system (Pujolras and Cairns, Reference Pujolras and Cairns2015). However, little research has been conducted to advance the theoretical understanding of how divergences in HTA outcomes might be explained. This paper compares cases of HTA in England and Germany using insights from Kuhn (Reference Kuhn1962) and Hall (Reference Hall1993) to demonstrate how policy paradigms can explain the outcomes of highly technical decision processes of significant political relevance.
The empirical evidence on which this paper draws concerns pharmaceutical benefit assessments, a common form of HTA, in England and Germany. Pharmaceutical benefit assessments are a particularly good set of cases for studying paradigms because ‘[…] elaborate paradigms are most likely to be found in fields where policymaking involves some highly technical issues and a body of specialised knowledge pertaining to them’ (Hall, Reference Hall1993, p. 291). Health care resource allocation through pharmaceutical benefit assessments involves such highly technical issues.
The first section of the paper provides an overview of HTA in England and Germany. This is followed by a discussion of the extant literature on divergent HTA outcomes, and a presentation of the analytical framework. An overview of the research methodology links these sections with the empirical section. The empirical findings show that pharmaceutical benefit assessment outcomes are shaped by different paradigms as HTA agencies engage in increasingly normalising patterns of decision-making, in which these paradigms are further refined. Applying insights from ideational accounts of the policy process demonstrates that the reasons for divergent and similar outcomes are more complex than structural accounts such as those based on path dependency, the set-up of political institutions and the degree of centralisation of the health care system would suggest (Löblova, Reference Löblova2016), and that policy paradigms are helpful theoretical tools to analyse empirical puzzles other than third order (Hall, Reference Hall1993) change. The policy and theoretical implications of the empirical findings are discussed in the last part of the paper.
2. Pharmaceutical benefit assessments in England and Germany
HTA is a scientific process in which the benefits, risks and frequently the cost effectiveness, of health technologies such as pharmaceutical products, are assessed by expert bodies to determine how they fare in comparison with routinely available treatments. HTA is informed by evidence such as multi-national RCTs, which pharmaceutical manufacturers submit to HTA organisations. It is classified as one of several policy instruments that are available ‘[…] to steer […] the use of medicines’ (Freeman, Reference Freeman, Marmor, Freeman and Okma2009, p. 247). As such, it is also conceptualised as a political process because its framework, methodology, stakeholder inputs, decision-making mandate and decision outcomes are ultimately shaped by political and social judgements of those involved in its processes (e.g. Abraham, Reference Abraham2003). It is distinct from the regulatory mechanisms of medicine licensing agencies such as the European Medicines Agency (EMA) in that it functions as an advisory mechanism for governments or public health insurance payers to determine health care entitlements. Several countries have established bodies that are tasked with conducting HTAs (Sorenson, Reference Sorenson2009), two of which, namely the National Institute for Health and Care Excellence (NICE) in England and the Institute for Quality and Efficiency in Health (IQWiG) in Germany, serve as case studies in this paper.
The National Health Service (NHS) in England is a public health care service, financed through taxation, and free for patients at the point of use. NICE was established in 1999 as an independent organisation to carry out HTAs to guide NHS decisions on which medicines to provide (Sorenson and Chalkidou, Reference Sorenson and Chalkidou2012). It conducts so-called technology appraisals (TAs) to assess the clinical and cost effectiveness of new pharmaceuticals. NICE makes one of three decisions: to recommend, not recommend or recommend only in research, the use of a medicine. Its decisions have a direct effect on what is funded on the NHS because health care commissioners are legally obliged to fund the drugs it recommends.
Germany's health care system is financed through statutory health insurance (SHI) and supplemented through taxation. Employers and employees pay health insurance contributions to quasi-public sickness insurance funds that finance and plan health services. The system is characterised by corporatist, self-governing organisations that plan health care and negotiate the terms of care provision, largely free from state interference (Kieslich, Reference Kieslich2012). The main decision-making body on minimum standards and level of health care is the Federal Joint Committee (FJC) (Perleth et al., Reference Perleth, Gibis and Göhlen2009). It carries out so-called early benefit assessments of new pharmaceuticals since 2011. In doing so, it commissions IQWiG to evaluate evidence presented by the pharmaceutical manufacturer. IQWiG acts as an advisory body to the FJC, with the FJC holding the final decision-making mandate. The clinical effectiveness of a new medicine is assessed according to its additional therapeutic benefit, of which there are six categories: major, significant, marginal, non-quantifiable and no additional benefit or a benefit that is less than the benefit of the appropriate comparator. In contrast to NICE, the FJC and IQWiG do not consider the cost effectiveness of a medicine, and the outcome of an early benefit assessment does not affect patient access to a medicine the way it does in England. The benefit category informs the price negotiations between the sickness insurance funds and pharmaceutical manufacturers. In theory, the higher the benefit category that a drug receives, the higher the starting price in negotiations.
3. Context and extant knowledge
This paper explores the question what determines pharmaceutical benefit assessment outcomes in countries with formalised HTA systems. It is situated in an evolving literature on the comparison of HTA outcomes in different countries, which has shown that HTA outcomes tend to differ between countries despite the same evidence in the form of RCTs being available. For example, Fischer et al. (Reference Fischer, Heisser and Stargardt2016) compared pharmaceutical benefit assessment outcomes in Germany, England, Scotland and Australia, and showed agreement between the FJC and NICE in only 40% of the final outcomes. A study by Nicod and Kanavos (Reference Nicod and Kanavos2012) found similar divergences, with 46% of the drug-indication pairs that were studied receiving diverging recommendations in five countries.
Several possible explanations for cross-country deviations exist, with the majority of the literature pointing towards multiple variables that shape assessment outcomes including institutional characteristics of the decision-making process (Böhm et al., Reference Böhm, Landwehr and Steiner2014) and industry-related factors (Abraham, Reference Abraham2009). Fischer et al. (Reference Fischer, Heisser and Stargardt2016) point to the differences in methodological guidelines of HTA agencies, with agencies accepting different endpoints, and different comparator products. Agencies also handle uncertainty around evidence differently, with some looking to other studies when evidence is uncertain and some, such as the FJC, concluding that the data provided by the manufacturer is insufficient. Nicod (Reference Nicod2017) confirms that differences exist in handling uncertainty in evidence, and in the way the same evidence is interpreted. She hypothesises that the reasons for these differences ‘[…] may have been influenced by agency-specific evidentiary, risk and value preferences, or stakeholder input’ (Nicod, Reference Nicod2017, p. 715).
Despite the evolving literature, the field remains under-researched theoretically in the sense that few attempts have been made to explain divergences applying theories about the public policy process such as idea- and agent-based approaches. In a systematic review of decision-making on health technologies, Fischer (Reference Fischer2012) argues that except for accountability for reasonableness (Daniels and Sabin, Reference Daniels and Sabin1997) no theoretical approaches have been applied to empirical research in this area. Since then, there has been some theoretically informed work on the diffusion of HTA processes in Europe (Löblova, Reference Löblova2016) and on the effect of institutional variables on HTA outcomes (Böhm et al., Reference Böhm, Landwehr and Steiner2014), but this work remains the exception rather than the rule in research on HTA processes. This paper seeks to address this gap by using insights from the ideational literature on policy paradigms to explain the outcomes of pharmaceutical benefit assessments in England and Germany.
4. Analytical framework
The analytical framework begins with the premise that values and ideas are important in priority-setting generally, and in HTA specifically. The premise arises from the literature on the ethics of health priority-setting and HTA (e.g. Biller-Andorno et al., Reference Biller-Andorno, Lie and Meulen2002; Norheim, Reference Norheim2002; Kenny and Joffres, Reference Kenny and Joffres2008) and on the role of social values (Clark and Weale, Reference Clark and Weale2012), which suggests that a value-free, objective and rational HTA process does not exist. Values, and ideas about how decisions should be made, underlie all HTA processes in an implicit or explicit way: ‘[…] it is important to be aware […] that any HTA is likely to be constrained by normative considerations, determining those facts to which we will turn our attention’ (Van Der Wilt and Reuzel, Reference Van Der Wilt and Reuzel1998, p. 35). The search for a theory was therefore guided by the need for a framework that accommodates the role ideas and values play in policymaking and decision-making.
The overview of the extant knowledge about divergences in HTA outcomes suggests that the interpretation of evidence, the determination of which forms of evidence to accept, and the way in which uncertainty is addressed are key contributors to HTA outcomes. With their emphasis on the influence of worldviews and paradigms that help determine what is or is not considered relevant in policymaking, ideational frameworks (Hall, Reference Hall1993; Béland, Reference Béland2005; Béland and Cox, Reference Béland and Cox2013) offer avenues to account for different interpretations of the same pieces of evidence. HTA systems are underpinned by a myriad of ideas, predominantly those rooted in evidence-based medicine. Other frameworks that take ideas into consideration exist, such as Sabatier's and Jenkins-Smith's (Reference Sabatier and Jenkins-Smith1993) advocacy coalition framework or Haas’ (Reference Haas1992) concept of epistemic communities. These approaches assume the collaboration of powerful, knowledge-based actors or groups that are bound together by belief systems that lead them to act in unity to achieve policy goals. For example, Löblova (Reference Löblova2018) demonstrates that epistemic communities in HTA are united by ‘[…] the conviction that HTA is an answer to problems of resource allocation in health care’ (p. 168). Given the aim of this research was to explore the factors that determine pharmaceutical benefit assessments, mapping these factors to the belief systems of advocacy coalitions and epistemic communities would follow as a more natural second step, going beyond the scope of the research at hand.
Although the role of ideas is now widely acknowledged as a variable in explaining policy change, their conceptualisation and operationalisation remains a challenge. Ideas have variously been described as encompassing ‘[…] everything from normative and ontological beliefs to perceptions about the disposition of other actors to understandings of causal relationships’ (Poteete, Reference Poteete2003, p. 532). However, the potential problems that arise from varying definitions are mitigated by the more widely accepted qualities of ideas as influential variables.
Ideas give rise to particular worldviews held by actors or groups in society. They influence which policy paths will be deemed conceivable in the context of a country's previous policy experience and institutional structure. This suggests that ideas have an overarching character, and that they are relational in nature (Hall, Reference Hall1989). Their relational character is important because it explains why the ‘[…] same ideas can be interpreted quite differently in settings where relevant historical experiences diverge’ (Hall, Reference Hall1989, p. 370). Thus, the second premise on which this paper rests is that ideas such as the ones associated with HTA take different forms in different countries, a premise that is supported in Hall's (Reference Hall1989) edited volume on the influence of Keynesian ideas across nations. A comparative study can help identify distinctive features of ideas, how they contribute to outcomes and how they interact with institutional factors. In this paper ideas are conceptualised as distinctive normative or empirical assumptions about policy problems, goals, tools and instrument settings (Hall, Reference Hall1993) as they relate to HTA policy.
Ideas influence policymaking in the form of policy paradigms, that is interpretive frameworks about the world, its problems and possible solutions: ‘[…] policymakers […] work within a framework of ideas […] that specifies not only the goals of policy and the kind of instruments that can be used to attain them, but the […] nature of the problems they are meant to be addressing […] this interpretive framework is a policy paradigm’ (Hall, Reference Hall1993, p. 279). The definition goes back to Hall's conceptualisation of the policymaking process as involving three variables: policy goals, instruments and instrument settings (Hall, Reference Hall1993). Policy goals refer to the overall goal that a policy aims to achieve, instruments refer to the choice of instrument to achieve said goal and the settings are the level at which they are set. To use Hall's (Reference Hall1993) example, if the goal was to alleviate old age poverty, policymakers might choose to implement an old age pension (the instrument) and the level at which the pension is set would be its settings. In HTA policy, the goal of the policy might be to ensure better value for money for the medicines that are covered in a health care system, the instruments by which to achieve the goal is the establishment of organisations such as NICE, and the settings might be the health economic methods used to assess medicines. The crucial point is that all three of these variables are influenced by a policy paradigm, a certain way of viewing the world. It follows that the three variables hold the key to the operationalisation of paradigms for the purpose of this study.
Before elaborating on the operationalisation of paradigms, a brief excursion on policy change is in order as policy paradigms are most often employed to explain significant policy change, a change that might signal a transition from one paradigm to another (e.g. Kay, Reference Kay2007; Menahem, Reference Menahem2008). However, Hall (Reference Hall1993) distinguishes three orders of policy change, only one of which represents paradigmatic change. First order changes are minor adaptations to instrument settings that do not alter the overarching ideas embedded in a paradigm, second order changes are changes in policy techniques and third order changes are substantial alterations in policy goals that may amount to paradigm shifts. Hall (Reference Hall1993) recognises various degrees of change. First and second order changes are more akin to ‘normal’ policymaking (Hall, Reference Hall1993), that is the process of making incremental adjustments in instrument settings and techniques as a policy is implemented. This process of ‘normal’ policymaking in which instrument settings are refined is a variable that is under-researched in ideational scholarship. This paper addresses this gap by focusing on the instrument settings of HTA policy to identify the parameters of HTA paradigms in England and Germany, thereby demonstrating how policy paradigms can explain empirical phenomena other than third order change.
The analytical approach is supported by Kuhn's (Reference Kuhn1962) theory of paradigm change in his study of scientific revolutions. He defined paradigms as ‘[…] a set of recurrent and quasi-standard illustrations of various theories in their conceptual, observational, and instrumental application’ (Kuhn, Reference Kuhn1962, p. 40). The process by which paradigms are transformed into ‘normal science’ is central, in other words the way in which paradigms are operationalised in practice: ‘[…] the study of normal-scientific traditions discloses many additional rules, and these provide much information about the commitments that scientists derive from their paradigms’ (Kuhn, Reference Kuhn1962, p. 40). By studying the rules to which scientific communities ascribe, the characteristics of a paradigm can be captured. The methods and instruments of scientific practice provide insights into these rules. Kuhn refers to this day-to-day activity of scientific practice as ‘normal science’, which ‘[…] means research firmly based upon one or more past scientific achievements, achievements that a particular community acknowledges for a time as supplying the foundation for its […] practice’ (1962, p. 10). Normal science can be thought of as paradigms in operation (Kuhn, Reference Kuhn1962, p. 11), thus by examining ‘normal’ practices, or ‘normal’ policymaking (Hall, Reference Hall1993), one gains insights into how paradigms are operationalised. To focus on paradigm change alone is essentially incomplete because to get to normal policymaking a paradigm needs instruments that have to be set in various ways. Studying the normalisation of paradigms is a prerequisite for distinguishing between true paradigm (third order) changes and incremental adjustments in policy.
It follows that HTAs are not an example of third order policy change waiting to be explained, but a good set of cases to learn about first order change, the process of how paradigms of pharmaceutical benefit assessments are refined in routine decision-making. The concepts of ‘normal’ policymaking and ‘normal science’ are adapted to the study of HTAs through the concept of ‘normalising decision-making’. The dependent variable of the study is the outcome of pharmaceutical benefit assessments. The focus is on the argumentative and logical processes by which HTA outcomes are explained in England and Germany, which are conceptualised as a reflection of how paradigms are normalised in practice. Although the analysis of the empirical material was largely informed by inductive reasoning, Hall's (Reference Hall1993) three variables – the techniques, goals and the instrument settings of a policy – were used as cognitive devices to organise the findings according to their relevance for the policy process. Since these three variables are informed by an overarching framework of ideas (Hall, Reference Hall1993), the policy paradigm, they were used as one way to capture a given HTA paradigm.
The second way to capture a given paradigm is through the process of normalisation. How the normalisation of paradigms takes place in public policy is poorly understood. In pharmaceutical benefit assessment policy, one way of conceptualising this normalisation is through ‘rules of evidence’: ‘When the issues under discussion require complex patterns of reasoning and large amounts of data of doubtful reliability and relevance, explicit rules of evidence become particularly important’ (Majone, Reference Majone1989, p. 10). Since assessments of pharmaceuticals depend on the availability and quality of evidence in the form of RCTs and other studies, they represent a set of cases that require complex patterns of reasoning. Through the development of rules of evidence in these processes, paradigms are further articulated. This paper will show that rules of evidence, akin to Hall's concept of instrument settings, can function as proxies to operationalise paradigms in highly scientific and technical policy areas. The importance of Majone's arguments ‘[…] is that it is the logical, not the institutional, features of scientific activity that provide the point of reference for the understanding of public policies’ (Weale, Reference Weale1992, p. 59). These logical, or argumentative, features of scientific (pharmaceutical benefit assessment) activity are the independent variables explored in this paper.
5. Methods and data
The study represents a small-N comparative case study with an embedded design. Two countries in which pharmaceutical benefit assessment systems exist, England and Germany, were compared by examining embedded units of analysis in the form of 10 pharmaceutical products. The country case selection was informed by relevance to the research aim (George and Bennett, Reference George and Bennett2005), i.e. to understand the determinants of pharmaceutical benefit assessment outcomes through the prism of paradigms. It focused on countries that employ institutionalised HTA processes at a national level. For reasons of comparability, the search was restricted to countries with public (tax-based) or SHI systems in Organisation for Economic Co-operation and Development (OECD) member states (Kieslich, Reference Kieslich2015), where the establishment of HTA organisations has been a popular policy instrument (Sorenson, Reference Sorenson2009).
Theory-driven considerations resulting from the novelty of employing policy paradigms to explain an empirical question other than third order change led a focus on the two health care systems still considered ideal-type health systems, that is the ‘Bismarck’ system in Germany and the ‘Beveridge’ system in England. These countries exhibit a number of institutional and historical differences from which different policy paradigms emerge. As such they represent particularly good cases for exploring how paradigms are articulated in different systems, with the aim to expand research to more settings in the future. An additional benefit was the comparable strength of the pharmaceutical sector, with both countries belonging to the top 10 pharmaceutical markets (IMS Health, 2015). This allowed controlling for the strength of the pharmaceutical market as a variable.
The embedded units of analysis were 10 cases of pharmaceutical benefit assessments (see Table 1). Medical devices and diagnostic procedures were excluded. The focus was on pharmaceutical benefit assessments so that evidence base and disease characteristics were controlled for. The 10 embedded cases were chosen using temporal criteria. In Germany early benefit assessments came into effect in January 2011, marking the starting point for the selection of the embedded cases. Considerations of feasibility meant that the endpoint for the search was August 2012. A total of 10 of the same pharmaceutical products were appraised by NICE, the FJC and IQWiG during this time (Table 1). These products constitute the units of analysis embedded in the case study. The generalisability of the findings is limited due to the small number of cases and embedded units of analysis. However, this limitation is justified by the aim to investigate the factors that determine benefit assessment outcomes through the prism of policy paradigms, an aim best achieved through an in-depth qualitative exploration of the data sources.
Table 1. Appraisal outcomes
Data sources included consultation and decision documents on the 10 embedded cases as well as statutory documents on HTA and pharmaceutical benefit assessments, HTA methods guidelines by NICE, the FJC and IQWiG, stakeholder position papers and stakeholder interviews. All sources are available online with the exception of the interviews that were carried out between October 2012 and June 2013 for the purpose of data triangulation. Twenty-three semi-structured interviews were conducted with stakeholders involved in the respective assessment processes of the embedded cases such as employees of HTA organisations, professional physician associations, patient advocacy groups and pharmaceutical manufacturers (see Table 2). The interviewees were identified using a purposive sampling method. Sixty interview requests were sent and 23 individuals responded positively. This translates to a successful response rate of more than a third. The interviews were anonymised and transcribed by the author.
Table 2. Interviewees and professional affiliations
Table 2 shows a balanced distribution in the number of interviews in England and Germany, but it also shows that the number of patient group interviews was higher in England than in Germany. The reason for this lies in the historical and institutional standing of these groups in Germany who are predominantly self-help groups with few financial or staff resources to become involved in HTA processes. Moreover, their involvement in the FJC is strictly regulated because patients are represented by a pooled patients' representation that declined to take part in an interview. Even with the institutional particularities of patient representation in Germany, the collected interview data were still useful because they contained views from individuals and groups who are involved in the decision-making process and who were able to provide insights into what determines HTA outcomes.
Della Porta's (Reference Della Porta, della Porta and Keating2008) concept of plural causation and George and Bennett's (Reference George and Bennett2005) concept of complex causality informed the qualitative content analysis. The analysis was informed by the idea that the configuration of factors, rather than their presence or absence, matters in contributing to given outcomes, and that similar factors can lead to different or similar outcomes. This is in line with others who have highlighted the benefits of comparative and case study approaches when the aim is to understand complex processes in which how variables matter might carry equal or more weight than whether they are present or absent in a causal process (e.g. Ragin, Reference Ragin1994; George and Bennett, Reference George and Bennett2005; Della Porta, Reference Della Porta, della Porta and Keating2008). The focus of the content analysis was on the reasons for final decisions in the embedded case studies. The data sources were read repeatedly, coded manually and inductively, from which themes around evidence and uncertainty (the rules of evidence), ‘special cases’ and external influences emerged. Hall's (Reference Hall1993) conceptualisation of the policy process was used to organise the themes of the empirical material and capture the respective paradigms. This organisation of the empirical material was supplemented using Majone's concept of ‘rules of evidence’ to identify the instrument settings of HTA policy.
6. Research findings
The data analysis indicates that pharmaceutical benefit assessment outcomes are determined by how a similar set of issues around evidence is interpreted by a HTA body. These issues are presented and discussed below. Unless otherwise stated, they emerged in each of the 10 embedded cases. Examples in the form of interview quotes or references to the HTA decision documents are drawn on to illustrate the emerging themes.
6.1 Core criteria, thresholds and coherence of paradigms
Each paradigm comes with a set of core criteria and thresholds that have an influence on whether different paradigms lead to similar or dissimilar outcomes. The core criteria can also be described as the thresholds that a medicine has to meet to attain a positive appraisal outcome. The prevalence with which thresholds were discussed and used as justification for decisions suggests that they are the expression of paradigms in the normalisation of decision-making as they suggest what is, or is not, possible. In England, the threshold is cost effectiveness: assessments are unlikely to result in positive recommendations if cost effectiveness criteria are not met. In Germany, medicines are unlikely to be assessed, or assigned a high benefit category, if clinical endpoints are not considered patient relevant.
The question of core criteria and thresholds centres on how cost effectiveness and patient relevance are operationalised, thus suggesting that core criteria can sometimes be indeterminate until operationalised. NICE operationalises cost effectiveness by using threshold ranges of incremental cost effectiveness ratios (ICERs) per quality-adjusted life year (QALYs). NICE's guide to TAs specifies that an ICER of £20,000–£30,000 per QALY will usually be deemed a cost effective use of NHS resources (NICE, 2013). However, over the years NICE has added criteria to its decision-making paradigm to better accommodate cases in which higher thresholds may be justified. These criteria function as modifying factors in HTAs, allowing the cost effectiveness paradigm to be stretched without challenging its core. For example, NICE's end-of-life criteria were introduced as a result of recurring situations in which end-of-life cancer treatments exceeded ICER thresholds, but were still perceived as valuable for the extra length of life they provided for patients, thereby necessitating an adjustment in instrument settings that marks an incremental change rather than a realignment of the paradigm.
In Germany, core criteria and threshold discussions centre on two questions. First, how patient relevance is defined and second, how the benefit categories are operationalised. In the case of Telaprevir, a treatment for chronic hepatitis C (Table 1), IQWiG questioned whether the surrogate clinical endpoint measured in the RCTs could be considered ‘patient relevant’. According to IQWiG, the sustained virological response (SVR) rate – that is whether the virus is still detectable in the bloodstream after recurrent laboratory checks – is a laboratory measure with no indication of whether a patient eventually develops complications, hence its patient relevance was questionable. The FJC disagreed with IQWiG and accepted the SVR as a patient relevant endpoint (G-BA, 2012h). In the case Fingolimod, a medicine for multiple sclerosis (MS), patient and clinical experts argued that its oral administration is a patient relevant consideration (G-BA, 2012). This was dismissed by the FJC as an outcome that could not be measured in clinical trials. The examples suggest a dissonance between decision-makers and stakeholders on how patient relevance should be defined and operationalised.
The second issue that is subject to discussions is the operationalisation of benefit categories. In an appendix to its first early benefit assessment IQWiG specified the thresholds between the six benefit categories in relation to confidence intervals and relative risk ratios (IQWiG, 2011). This approach has been included in an updated version of IQWiG's methods guidelines (IQWiG, 2015). However, in the majority of the cases studies, the FJC states that: ‘The method proposed by IQWiG in Appendix A of the benefit assessment dossier on Ticagrelor […] was not relied upon […]’ (e.g. G-BA, 2012b, p. 3). Both IQWiG's method and the FJC's reluctance to follow it give rise to criticism by stakeholders. The former is criticised for not being scientifically validated, and the latter is criticised for not being transparent about how benefit categories are operationalised. A FJC representative stated: ‘[…] I can't say that […] the FJC has developed a matrix for making its decisions. […] I […] don't know whether our system is earmarked for this, for structuring something in such a mathematical way that it becomes predictable […]’ (Representative of HTA organisation, 2013).
These examples of the process of normalisation suggest a reluctance to operationalise benefit categories by means of algorithmic methods, a distinctive difference from the cost effectiveness paradigm under which NICE operates. The controversies also suggest within-paradigm contradictions, and dissonances between the stakeholders involved in the assessment process. The contradictions arise from IQWiG's operationalisation of patient relevance as the endpoints of clinical trials, in other words ‘hard’ outcome measures, which is contrasted by the FJC's apparent reluctance to subscribe to a paradigm based on thresholds that make the outcomes of its decision processes predictable.
The controversies over the definition and operationalisation of patient relevance suggest that not all stakeholders perceive the paradigm as a coherent construct. By contrast, the operationalisation of the cost effectiveness paradigm in England, whilst not always resulting in outcomes for which stakeholders hope, was not the focus of criticisms. However, the apparent coherence of a paradigm does not mean that it is immune to change, as illustrated by the introduction of NICE's end-of-life criteria. The introduction of end-of-life criteria exemplifies a case of first order change in which decision-makers adjust instrument settings in response to new challenges.
6.2 When paradigms are challenged
The way in which paradigms are adjusted when challenges emerge offers revelations on how they operate in practice. Evidence presented in HTA processes frequently gives rise to uncertainty about the quality of studies. As a result, HTA decision-makers make judgements on how rules of evidence should be operationalised. Some instances of uncertainty appear to be more challenging than others, particularly where ‘special’ cases are concerned, referring to cases in which decision-makers deliberate on a unique set of challenges during the appraisal of evidence on medicines for chronic and long-term conditions. This was the case for Fingolimod and Retigabine (see Table 1). Medicines for conditions such as epilepsy have the potential to challenge a given paradigm if current rules of evidence do not provide answers to the questions raised.
In the case of Retigabine, a medicine for the treatment of epilepsy (Table 1), clinical experts pointed to the limits of RCTs, commonly perceived as the gold standard of evidence-based medicine. RCTs are designed as short-term clinical studies that measure outcomes whilst controlling for as great a number of factors as possible to show causality between a given outcome and the administered drug. However, chronic diseases such as epilepsy and MS are complex, they present differently in every patient, and their disease progression is difficult to predict. This presents challenges for designing RCTs. For epilepsy, one clinical expert outlined the following challenges:
[…] 50% seizure reduction […] doesn't […] have a lot of […] weight […] it would be like jumping from the 5th floor instead of the 10th […] they [the outcomes] have no clinical meaning, they are ok to convince the FDA and the EMA […] but […] what I'm looking for is seizure freedom (Specialist Consultant, 2013).
NICE accepted that there were limitations to the data on Retigabine, and heard from clinical and patient experts that seizure freedom would be a more valuable outcome than 50% seizure reduction (NICE, 2011a). NICE balanced the available evidence from the clinical trials and the evidence from experts, and recommended Retigabine in patients where other treatment options had failed. IQWiG and the FJC did not consider the available evidence in the first place because the trials did not compare Retigabine to the appropriate comparator (G-BA, 2012g). As a result, the FJC concluded that an additional benefit could not be substantiated regardless of clinical experts arguing that Retigabine provides a useful, and patient relevant, treatment option where other treatments have failed. This suggests a degree of risk aversion by German HTA administrators to stretch the parameters of the paradigm of patient relevance, which in practice demands clinical studies that directly compare the treatment in question with its appropriate comparator.
6.3 Rules of evidence
Rules of evidence, e.g. how evidence is conceptualised, the quality of the evidence, how best to separate patient groups, and how to determine the comparator products, formed a significant part of the argumentative process in all of the examined cases. In the case of Fingolimod NICE, IQWiG and the FJC highlighted that the patient population included in the RCT was larger than that for which a marketing license had been granted (G-BA, 2012; NICE, 2012). This means that the patient population that is eligible for treatment is smaller than the one on which it has been tested, thereby necessitating an investigation of the applicability of trial results to the patient population in routine clinical practice. To address the incongruence, NICE accepted indirect comparisons to extrapolate an estimate of Fingolimod's expected benefits. It heard from patients and clinicians that the oral administration of Fingolimod is an innovation as previous treatment options for MS patients came in the form of injections (NICE, 2012). NICE issued a positive recommendation for Fingolimod as an option for the treatment of highly active relapsing-remitting MS.
IQWiG and the FJC did not accept any indirect comparisons of data. There were contrasting opinions between IQWiG and the pharmaceutical manufacturer on what constitutes ‘best available evidence’. While the manufacturer argued that indirect comparisons constituted best available evidence, IQWiG argued: ‘[…] the best available as well as the best possible evidence for a decision problem – must […] be suitable for answering a […] problem. If this is not the case […] the […] best available evidence […] is irrelevant. This is the case for the indirect comparisons […]’ (2012, p. 51). The FJC assigned a positive (marginal) added benefit to the patient group with rapidly evolving severe relapsing-remitting MS, only one of several potential patient groups eligible for treatment with Fingolimod. NICE recommended the use of Fingolimod for all patient populations except for those patients with rapidly evolving severe relapsing-remitting MS, whereas the FJC concluded that the available evidence showed an additional benefit for only this patient group.
The case exemplifies that instrument settings of paradigms, such as the distinction between ‘best available’ and ‘best possible’ evidence, are set during normalising decision-making processes. The settings are connected to the way in which HTA decision-makers operationalise hierarchies of evidence, and prior to operationalisation the settings are largely indeterminate. NICE, FJC and IQWiG method guidelines stipulate a preference for RCTs in their hierarchies of evidence (G-BA, 2013; NICE, 2013; IQWiG, 2015). In the absence of good quality RCTs, they allow for other forms of evidence to be considered, yet in practice the FJC and IQWiG are reluctant to consider indirect comparisons. Rules of evidence are refined in practice, which leads to paradigms operating in different ways despite similarities in documents such as HTA method guidelines.
6.4 Political power, public pressure and stakeholder influence
The role of political power, public pressure and stakeholder influence did not emerge in all of the case studies, but rather in England in the case of Abiraterone. As such, Abiraterone could be considered an outlier case. However, given the significance of the issues it is crucial to report on them and compare them with the situation in Germany as they provide insights into the interplay between power and ideas, thus opening avenues for future research.
For Abiraterone, an anti-hormonal therapy for prostate cancer, NICE initially issued an Appraisal Consultation Document (ACD), a document that is published if the recommendation is negative to allow for a consultation process (NICE, 2012g). Following the ACD there were protests in the form of media campaigns by patient groups. NICE eventually reversed its position on grounds that the manufacturers had submitted further evidence (Edgar, Reference Edgar2012).
While no definite causal inferences can be drawn between the public campaigns and NICE's ultimate decision, not least because the final decision was still justified using cost effectiveness criteria, interviewees suggested that public campaigning was at least a contributing factor. A representative of a charity involved in the campaign stated:
[…] the [appraisal] committee are […] rigid […] if they don't see the evidence […] then they will […] continue to say ‘no’ but […] what we did […] was to communicate […] to the media the case around Abiraterone and why it should be made available […] we did make a difference here […] it made them [NICE] think again […] (Charity representative, 2013).
The statement that evidence is key to a positive appraisal outcome suggests an acknowledgement that public campaigns are more likely to be successful in reversing negative decisions if they appeal to core criteria such as cost effectiveness thresholds.
Early benefit assessments receive little media and public attention in Germany. However, questions of influence emerge with regard to the bargaining powers of the statutory sickness insurance funds. The statutory funds are involved in every step of the pharmaceutical assessment process. As an umbrella organisation the Federal Association of Statutory Sickness Funds is a member of the FJC. It contributes to the decision on the appropriate comparator, it is involved in assessing the benefit category, and finally it leads the price negotiations with pharmaceutical manufacturers. This has led to a perception of the sickness funds as having a stronger bargaining position: ‘[…] we have a classic monopolisation due to the fact that it [the Federal Association of Statutory Sickness Funds] always negotiates, it has extreme learning curve effects […]’ (Pharmaceutical industry representative, 2013).
In comparison with the public pressure that NICE faces, the controversies over the bargaining positions in Germany are less visible. This is a reflection of different institutional structures, with the German health system functioning as a corporatist, self-governing system. The limited data that emerged on this point indicate that, so far, NICE's paradigm is equipped to absorb political and public pressures by accepting further information in the process of normalising decision-making. However, recent changes in NICE's approach to medicines appraisal and its consideration of budget impact (Charlton et al., Reference Charlton, Littlejohns, Kieslich, Mitchell, Rumbold, Weale, Wilson and Rid2017) suggest that this may be changing.
7. Discussion
The issues that influence pharmaceutical benefit assessment outcomes can be distinguished between issues that are logical, that is issues that require reasoning within the paradigm so as to define how to set the instruments, and issues that are institutional and interest-based. The issues that require interpretation are similar sets of issues around evidence that are resolved during the process of normalisation. The majority of the findings pertained to instrument settings, i.e. the rules of evidence, of the respective HTA policies, which were used as proxies to understand how the HTA paradigms operate in practice. This is in line with the current literature on divergences in HTA outcomes (Fischer et al., Reference Fischer, Heisser and Stargardt2016; Nicod, Reference Nicod2017). The issues that are institutional and interest-based are the openness to public or stakeholder pressure and political power. A closer look at how these sets of issues interact with one another allows for an appreciation that paradigms can be largely indeterminate until they are operationalised.
The logical issues that emerged at the instrument setting level indicate that NICE's paradigm of cost effectiveness functions in a more flexible way than the paradigm of patient relevance in Germany. HTA decision-makers in Germany appear more risk averse in stretching the core of the paradigm, a finding that has been discussed in extant research (Fischer et al., Reference Fischer, Heisser and Stargardt2016). The apparent risk aversion can be explained with reference to the goals of HTA policy. The policy goal of HTAs in England is the recommendation for inclusion or exclusion of a treatment in the NHS health benefit basket. In Germany, the goal is to inform the price negotiations between sickness insurance funds and pharmaceutical manufacturers. This means that the consequences of negative decisions in England and in Germany are different. In England, a negative decision will result in the treatment not being available in routine care. In Germany, the treatment will still be available, but the industry's negotiating starting point is not as strong as with a positive decision. Put simply, HTA decision-makers in Germany can afford to be less flexible, and more risk averse, when adjusting the paradigm of patient relevance because their policy remit, and their impact, is restricted. As such the incentive, or the need, to refine the paradigm in normal decision-making is less pronounced than in England. By comparison, NICE's recommendations are mandatory, and arguably an inflexible interpretation of the paradigm in practice would be unsustainable given the effects that would emerge. This shows how instrument settings interact with policy goals within a given policy area.
The role of political power, public pressure and stakeholder influence in determining HTA outcomes speaks to the interaction between ideational and interest-based variables. It thus maps onto the territory of the influence of beliefs and interests found in the literature on public and patient involvement in health priority-setting (e.g. Weale et al., Reference Weale, Kieslich, Littlejohns, Tugendhaft, Tumilty, Weerasuriya and Whitty2016), the sociology of pharmaceuticals (Abraham, Reference Abraham2003) and work on epistemic communities in HTA diffusion (Löblova, Reference Löblova2018). The interaction between ideational and interested-based variables is important as paradigms are too often discussed as frameworks of interpretation without reference to the actors that interpret them. Actors are important because the agents operationalising a given paradigm through setting instruments are administrators, bureaucrats and experts who have their own set of experiences, values and motivations. Future studies of HTA paradigms would benefit from investigating who the agents in the normalisation process are, and how their belief systems map onto the ideas of HTA paradigms.
The motivation of stakeholders, including politicians, to get involved and exert pressure on HTA institutions differs in England and Germany, largely as a result of different institutional structures that offer distinct veto points for intervention (Hall, Reference Hall1989; Immergut, Reference Immergut1992). In Germany, the self-governing actors in the health care system have political, financial and institutional resources that allow them to influence decisions by being directly involved in the HTA decision-making processes. The strength of the self-governing health care regime is reflected in the HTA policy goal that HTA recommendations are to determine price negotiations. Embedded in this institutional set-up is a commitment to the idea that the self-governing actors, and not the state, determine how to allocate resources in health care. Such institutional characteristics are important considerations when distinguishing between incremental and paradigmatic change. Paradigms prevail as long as they provide answers to the dominant questions of the day. Political power, public pressure and stakeholder influence are variables that, in time, may lead to paradigmatic change if HTA decision-makers are unable to refine the paradigm in a way that satisfies the most powerful actors in the system.
The case studies confirm Hall's (Reference Hall1993) argument that first order change happens frequently, and can be conceptualised as incremental policy change that is to be expected during the process of implementation. Paradigms are fluid, and rather indeterminate, constructs that undergo constant refinement. By focusing on paradigms as they operate in normalising decision-making, the parameters of the paradigm become more nuanced. For example, the FJC and IQWiG's conceptualisation of the criterion of patient relevance is narrower than the connotations of the term would have observers, and indeed stakeholders, believe. In the case of Fingolimod, the FJC argued that the oral way of administering the drug was not patient relevant because it could not be measured as a clinical endpoint. It illustrates the usefulness of investigating the instrument settings of a given policy because the researcher learns that HTA decision-makers conceptualise the criterion as closely related to clinical outcomes. This is an operational detail that would have been missed without reference to the process of normalisation.
8. Conclusion
The paper illustrates how policy paradigms can be used as analytical frames to explain pharmaceutical benefit assessment outcomes, thereby advancing the theoretical explanations of divergent HTA outcomes, and the application of policy paradigms to empirical puzzles other than third order change. This offers opportunities to better distinguish between paradigmatic and incremental policy change in HTA. For policymakers, the public, patient organisations, and the pharmaceutical industry, a better understanding of the factors that influence benefit assessment outcomes opens up opportunities for policy adjustment and influence. Policymakers might observe trends in HTA processes that are contrary to the values they wished to embed in setting up HTA bodies. If that is the case, political action can be taken to make changes. Stakeholders such as pharmaceutical manufacturers and patient organisations can adjust their communication strategies based on the lessons learned on how evidence is interpreted through different paradigmatic lenses.
The usefulness of policy paradigms in explaining HTA outcomes has implications related to current developments in assessments of very expensive medicines. Political debates have ensued in England and Germany, and around the world, about new medicines for hepatitis C, such as sofosbuvir (Kieslich et al., Reference Weale, Kieslich, Littlejohns, Tugendhaft, Tumilty, Weerasuriya and Whitty2016). These medicines are equated to a cure, but they come at a high price (Gornall et al., Reference Gornall, Hoey and Ozieranski2016). In England, sofosbuvir was appraised to be within NICE's cost effectiveness threshold range, yet debates about the affordability of the medicine for the health system continue (Gornall et al., Reference Gornall, Hoey and Ozieranski2016). The challenge to existing HTA paradigms is that their core values (cost effectiveness and patient relevance respectively) may no longer be able to address the policy problem, i.e. how to balance cost effectiveness and patient relevance with affordability. According to Hall (Reference Hall1993) and Kuhn (Reference Kuhn1962) such instances, where available instruments no longer address the problems of the day, open up the opportunity for a paradigm shift. The findings on HTA paradigms provided here offer an analytical basis on which to assess the significance of recent changes in HTA policy such as the introduction of NICE's ‘budget impact test’ (Charlton et al., Reference Charlton, Littlejohns, Kieslich, Mitchell, Rumbold, Weale, Wilson and Rid2017).
