Health technology assessment (HTA) conducted to inform developers of health technologies is typically characterized in the academic literature as “early” health technology assessment. We prefer the label “development-focused HTA” (DF-HTA) as is it explicitly describes the purpose of the analysis, rather than alluding to just one, arguably not defining, characteristic.
DF-HTA has a number of distinct features when compared to HTA conducted to inform reimbursement and usage decisions (use-focused HTA). To conduct effective DF-HTA, it is important that analysts, who are often more familiar with use-focused HTA, are made aware of these differences. This is particularly important as DF-HTA analyses conducted or commissioned by commercial technology developers and in-house analyses undertaken in pharmaceutical and large medical device companies are typically not published due to a desire to maintain confidentiality and lack of incentive to publish (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1). Assessments of medical devices developed by small and medium enterprises (SMEs) or academic groups may be published if some public funding has been provided. Consequently, the studies which are published are likely to be a biased sample of the work undertaken and may not be particularly useful as a reference source for HTA analysts new to working with developers.
There is some useful methodological content in the published literature. For example, the Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH) collaboration in the UK aimed to support companies in the UK healthcare technology sector to assess the value of medical devices from concept through to mature product. The collaboration extended the concept of iterative economic evaluation described in the late 1990s and 2000 (2;3) to develop methods and tools (e.g., Reference Cosh, Girling, Lilford, McAteer and Young4–Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8) for SMEs. The SMEs were often working in resource-constrained environments and had little in-house knowledge or experience of HTA (Reference Craven, Allsop, Morgan and Martin9). Other groups, particularly translational research bodies such as ProHTA (Reference Kolominsky-Rabas, Djanatliev, Wahlster, Gantner-Bär, Hofmann and German10) and the Center for Translational Molecular Medicine (Reference Steuten11), built upon the MATCH work and published further methodological and applied papers (e.g., Reference Ijzerman and Steuten12–Reference de Graaf, Postmus, Westerink and Buskens14). This form of HTA, initially described as “supply side” HTA by McAteer et al. of the MATCH collaboration (Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8), has more recently been described as “early HTA” (12;15).
We find the term “early HTA” as a term to describe DF-HTA somewhat vague and unhelpful. It might be taken to imply that methods and approaches for DF-HTA are essentially similar to those of the more commonly reported use-focused HTA, only undertaken at an earlier point in time. This problem is compounded by the fact that many early health economic modeling studies in the published literature take the normative structure of use-focused HTA and apply it at an earlier point in time. However, we argue that there are fundamental differences between development- and use-focused HTA that arise as a consequence of the differences in the target audience and the decisions that the analysis is intended to inform and that these are more important than the timing of the analysis.
In this paper, we aim to produce a characterization of DF-HTA that is useful to analysts new to working in this field. The framework is intended as an aide-memoire for analysts more familiar with use-focused HTA. We suggest that it is used at the outset of a project to prompt reflection on the nature of the assessment and to help structure discussions with developers. It can also be used at the reporting stage of a project, whether published or not, to ensure transparency. We hope that the suggested framework forms the starting point for a debate in the wider academic community about the nature of HTA undertaken to inform developers of health technologies.
To develop the framework of features of DF-HTA, we set out an initial framework based on our existing experience and a review of published literature. The literature review used a pearl-growing approach (Reference Claxton, Martin, Soares, Rice, Spackman and Hinde16) which aimed to identify examples of applied and methods papers where the intended audience was developers of medical technologies. We used Ijzerman and Steuten's 2011 review article (Reference Ijzerman and Steuten12) as the initial pearl. This approach involves reviewing the references and citations of the pearl for articles of interest, then reviewing the references and citations of the articles of interest until saturation is reached. The search was undertaken in October 2017 and refreshed in February 2019.
Features of Development-Focused HTA
Pietzsch and Paté-Cornell in 2008 (Reference Pietzsch and Paté-Cornell15) and Ijzerman and Steuten in 2011 (Reference Ijzerman and Steuten12) identified four features distinguishing early HTA from “Classical” or “Mainstream” HTA as they termed HTA undertaken to inform decisions taken at the market access stage. These features were: target audience; decisions to be informed; available evidence; and, timing. In addition to these four features, we identified a further four features in our literature review. These were: underlying user objective; decision space; business model; and resources for analysis. We then added a further two features of DF-HTA based on our experience: stance of analysis and burden of proof. The ten features are presented as a proposed framework in Table 1. These identified features are not separate and independent but intimately linked. For example, evidence is lacking because of the timing of the assessment. However, they are each worthy of explicit consideration.
Table 1. Features of DF-HTA
Target Audience
The target audience for DF-HTA includes both the developers of technologies and the sponsors or funders of the development. These may include both commercial and academic institutions as well as private and public sectors funders (Reference Pietzsch and Paté-Cornell15). We will hereafter use the general term “developers.” The target audience of developers is, in our view, the defining feature of this form of HTA. DF-HTA differs from other forms of HTA because of the requirements of its target audience. In published studies, the target audience is often not explicitly defined (e.g., Reference Hjelmgren, Ghatnekar, Reimer, Grabowski, Lindvall and Persson17–Reference Vilsbøll, Mouritsen, Jensen, Bødker, Holst and Pennisi19). In some cases, the analysts appear to adopt the perspective of a payer even when the HTA is undertaken to inform the developer. For example, Latimer et al. (Reference Latimer, Dixon, McDermott, McCarthy, Tindale and Heron20) undertook an economic evaluation to inform developers about the feasibility of designing a collar for use by patients with motor neurone disease which would be cost-effective from the perspective of UK NHS. Such an analysis fails to explicitly recognize that the technology might be marketable in multiple markets that apply differing criteria to determine reimbursement.
Underlying User Objective
The primary objective for a commercial sector developer or investor is to maximize long-term financial return on investment (12;21–23). Other social objectives or motivations are typically subservient to this objective. The primary objective of public sector developers (e.g., academic developers funded by public bodies) is to maximize the societal return on investment. Societal return includes consideration of direct financial returns on development, industrial growth or employment, and improvements in societal health [e.g., Innovate UK's funding streams (24)].
To maximize long-term financial return on investment, developers and investors need to consider the measures of value for money that payers use in their coverage decisions. Thus, the underlying objective of the payer is relevant for developers to inform pre-clinical, preliminary market assessments, and first estimations of pricing and reimbursement scenarios. Although explicit thresholds, such as the £20,000 to £30,000 per quality-adjusted life-years in the UK (25), are often used in DF-HTA to make a first estimate of the maximum price achievable for a technology in order for it to be considered cost-effective (7;8;26) it is important for DF-HTA analysts to recognize that a range of approaches are used by different payers.
The underlying decision rules used by decision makers should reflect their objectives. Analysts undertaking DF-HTA should acknowledge that commercial developers and investors will use, either implicitly or explicitly, a decision rule based on the expected net present value of an investment. This means that they will be interested in the expected revenues to be generated across relevant markets and the expected costs associated with delivering these sales as well as the timescales over which the revenues and costs occur. In principle, the net present value of these potential inflows and outflows should be calculated using a discount rate which takes into account the company's cost of borrowing reflecting the perceived risk of the project. In practice, crude measures of the opportunity costs of a particular investment are likely to be used.
Public funders and non-commercial developers may base a decision to continue the development on a formal net value of information analysis based on the acceptable cost-effectiveness threshold (which itself should represent the opportunity cost of healthcare expenditure). More informal analyses may simply try to estimate the likelihood or potential that a technology will be regarded as cost-effective. In this situation, the decision rule used is to continue the development if the technology is likely to meet the appropriate thresholds for cost-effectiveness in the relevant jurisdiction. Commercial developers and investors may also be interested in the outcome of this analysis as it would provide some indication of the likelihood of sales in the relevant jurisdiction and provide guidance as to acceptable pricing. It should be recognized that explicit thresholds are not used by all payers, they vary between jurisdictions, they are not the only determinant of reimbursement, and they are subject to change. Commercial developers and investors may also take into account other aspects of value not typically included in formal cost-effectiveness analyses, such as patient convenience or comfort and the value of knowing a diagnosis for patients and their families, if these are likely to influence usage (18;27).
Decisions HTA Designed to Inform
DF-HTA potentially informs a wide range of decisions and considerations including: preliminary market assessment; estimation of pricing; review of reimbursement scenarios; individual go/no go decisions; technology design; evidence generation strategy including study design; and research and development portfolio prioritization (15;28). As DF-HTA is undertaken before the development process concludes, developers can respond to the assessment by changing the design of the technology, its target indication(s) and position in the clinical pathway (Reference Pietzsch and Paté-Cornell15). The assessment process itself may highlight gaps in the evidence for the new technology which can drive the evidence generation strategy at the next phase of development. This can also facilitate discussions with regulators or reimbursement agencies that increasingly offer to engage with developers during the development process. If assessment is undertaken simultaneously for a number of technologies, the results can be used to identify the most promising technologies facilitating the prioritization of research effort and expenditure. For example, de Graaf et al. (Reference de Graaf, Postmus, Westerink and Buskens14) assessed the potential of biomarker tests in four roles in the prevention of type-2 diabetes mellitus to prioritize research effort and expenditure within a translational research organization.
Decision Space
By decision space, we mean the range of different ways and places in which a technology may be used, for example, clinical indication, target population, and placement in the treatment pathway. In DF-HTA, the decision space is often wide and poorly defined. As DF-HTA is generally undertaken prior to licensing, the potential indications and positions in the clinical pathway are not yet constrained by licensing restrictions and multiple options may need to be assessed (Reference Annemans, Genesté and Jolain3). Other aspects of decision space include multiple versions of the technology (including optimization of test characteristics for diagnostics) (Reference Buisman, Rutten-van Mölken, Postmus, Luime, Uyl-de Groot and Redekop23), patient populations (11;21), jurisdictions, comparators, dosages, modes of delivery, pricing structures (Reference Annemans, Genesté and Jolain3), and diffusion scenarios (Reference Rogowski, John, Ijzerman and Scheffler27). Furthermore, these may vary across different potential markets.
Available Evidence
In DF-HTA, evidence specific to the technology is typically scarce early in the development process. As direct evidence of clinical effectiveness is lacking there is more reliance on elicited expert opinion (4;21), evidence relating to comparator technologies (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton21), bench or animal studies, previous generations of a technology (Reference Pietzsch and Paté-Cornell15), and extrapolations from pharmacodynamics models (Reference Annemans, Genesté and Jolain3). Where direct clinical evidence is available, studies are often small so that uncertainty around any estimates is high. Methods of expert elicitation have been developed to improve the reliability of experts' estimates of plausible ranges. Evidence may also be required about usability or the impact of a technology on clinical pathways (Reference Abel, Shinkins, Smith, Sutton, Sagoo and Uchegbu29). Qualitative methods (30;31) and multi-criteria decision analysis (Reference Hummel, Van Rossum, Verkerke and Rakhorst32) have been used to address this need. Shortage of evidence is not unique to DF-HTA, as uncertainty is inherent in all HTA. However, the shortage is likely to be more pronounced earlier in a development process.
Timing
We see DF-HTA as an ongoing activity facilitating a continuous discussion around the technology development process rather than a discrete event with a specific output. The majority of DF-HTA will be undertaken before a technology is approved by a regulatory body. The starting point for the DF-HTA may be the identification of a clinical need preceding the product development process (Reference Yock, Zenios, Makower, Brinton, Kumar and Watkins33). In this case, the DF-HTA would assess the potential for the technology proposed. An example of this approach is provided by Brandes et al. (Reference Brandes, Sinner, Kääb and Rogowski34), who assessed a hypothetical vascular closure device and found only a single sub-group where the technology had potential. Alternatively, the starting point for DF-HTA may be the evaluation of a technology already in development. Kluytmans et al. (Reference Kluytmans, Tummers, van der Wilt and Grutters31) evaluated a surgical device at prototype stage and found that there was little potential for the device in meniscus surgery, which was the developers' suggested indication. DF-HTA is particularly suited to an iterative approach with discussions with developers continuing alongside the development process and analysis undertaken prior to significant investments, such as Phase II or Phase III trials for pharmaceuticals (5;6). Vallejo-Torres et al. (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton21) presented an iterative economic evaluation of absorbable pins for hallux valgus at three different stages of development. The authors used retrospective data for this analysis to recreate the dynamic process of DF-HTA occurring in real-time alongside the development process. It should also be noted that use-focused HTA may also use an iterative approach (rather than the discrete event with a specific output described above), as products are arriving to market with greater levels of uncertainty.
Business Model
In this context, the term “business model” broadly refers to how a technology and the customer are brought together, which determines how the revenue stream is generated and what barriers there may be to entry (Reference Rogowski, John, Ijzerman and Scheffler27). In DF-HTA, the business model may not be fixed. Developers have the option to offer their technology (subject to local regulatory constraints) wherever the potential is greatest and to target patients and/or clinicians directly or to sell via national health services. For example, van Nimwegen et al. (Reference van Nimwegen, Lilford, van der Wilt and Grutters18) used parents' willingness to pay for a diagnosis to calculate “headroom” (valuing an estimated extension in life and/or improvement in quality of life at a given threshold value with an adjustment for the cost impact of the technology) rather than an explicit threshold for reimbursement as it was felt that the technology would be best suited to the private payer market. The business model adopted by the commercial developer or investor may differ across jurisdictions. Non-commercial developers may also need to consider commercial means of bringing their technology to market, as established biotech companies maybe best-placed to maximize the technology's potential.
Resources for Analysis
In the early stages of development, in large companies, there may be a set of candidate technologies which could potentially be assessed using DF-HTA. As many of these potential technologies will fail (3;5) resource-intensive approaches to HTA themselves may not have a positive expected net present value. DF-HTA must compete for scarce resources, potentially displacing aspects of the research and development process. In addition, many medical devices, including tests, are developed by small- and medium-sized enterprises and may be the sole product of that company (Reference Craven, Allsop, Morgan and Martin9). Such companies may have limited HTA experience and resources. This means that DF-HTA must deliver value within significant resource constraints. At the earliest stages of development, it is suggested that effort is focused on articulating and quantifying a value proposition (5;27). This could potentially be done using qualitative interaction with clinicians and users (30;31) and simple quantitative methods such as headroom analysis (Reference Chapman7). This prima facie case can then be developed further as the development progresses when more resources may be available (5;23;27).
Stance of Analysis
By stance of analysis, we mean the mindset adopted by the analyst in undertaking the assessment. We believe that the adoption of a positive rather than a normative economic stance of analysis is one of the fundamental features of DF-HTA, which has not previously been widely discussed. DF-HTA for commercial developers adopts a positive stance, as no value judgments are required (Reference Culyer35) and the analysis is focused on the maximization of the developers' return on investment. For example, Hummel et al. (Reference Hummel, Boomkamp, Steuten, Verkerke and Ijzerman36) mentioned that the aim of their analysis was to “support the future development” of the technology. Similarly, Kluytmans et al. (Reference Kluytmans, Tummers, van der Wilt and Grutters31) commented that much early HTA “has a strong technology-focused or supply-driven character.” We concur with this statement as, in our experience, developers start with the technology and part of the role of DF-HTA is to find a place where it can be successful. In this sense, DF-HTA has the character of a formative assessment, that is, an assessment to further the development. By way of contrast, use-focused HTA has the character of a summative assessment against a pre-determined set of criteria. Use-focused HTA adopts a normative stance; it involves judgments about what is good for society (Reference Culyer35).
Burden of Proof
There are no guidelines about either methods to be adopted or the acceptable level of evidence required for DF-HTA, nor would such guidelines be appropriate. The process of DF-HTA is iterative; initial stages use whatever evidence is available and methods deemed appropriate by the analyst. The output from the HTA process informs the discussion between the developer and the analyst and takes any limitations in evidence and methods of assessment into account. For use-focused HTA, in many jurisdictions, there are clear guidelines as to what level and form of evidence the reimbursement agency or payer deems acceptable as well as how the assessment should be undertaken. For example, the National Institute for Health and Care Excellence, the reimbursement agency for England and Wales, prefers the evidence of health effects to come from randomized-controlled trials directly comparing the intervention with one or more relevant comparators and has a comprehensive guide to methods (25).
How the Framework may be Used
We suggest that our framework of features of DF-HTA is used as an aide-memoire at the planning stage of a project, in initial discussions with developers. This would help to clarify essential features of the analysis in the mind of the analyst and ensure transparency between the developers and the analyst. Certain features of the framework may encourage discussions about features which would be unlikely to be discussed otherwise, such as the developers' underlying objective. It may also encourage a consideration of the wider decision space or alternative business models. Additionally, the framework could be used as a checklist for reporting to developers or in a published article to ensure that the characteristics of the analysis are transparent. Table 2 shows a summary of questions for consideration or discussion.
Table 2. Questions for consideration in DF-HTA
Discussion
We set out to provide a characterization of HTA undertaken to inform developers. We described ten features of DF-HTA in a framework to be used as an aide-memoire for analysts new to this work and as a checklist for reporting. Four of the features (target audience, decisions to inform, available evidence, and timing) had been included in previous frameworks distinguishing early and mainstream HTA (Reference Ijzerman and Steuten12) or classical HTA (Reference Pietzsch and Paté-Cornell15). The remaining features (underlying user objective, decision space, business model, resources available for analysis, stance of analysis, and burden of proof) were identified by the authors, informed by our experience and the methodological and applied papers identified in our review.
Although previous authors have gone some way toward characterizing DF-HTA (as part of “Early HTA”) (12;13;15), it was often conflated with other activities where evidence was scarce such as horizon-scanning (Reference Ijzerman and Steuten12) and the assessment of process or innovation from the perspective of the health service provider (13;37). Although these related activities may share some features with DF-HTA such as the timing and the lack of evidence, they differ significantly in important aspects of the work. In particular, the target audience for the work is healthcare decision makers and the stance of analysis may be normative in nature. Authors associated with the MATCH collaboration in the UK set out a methodology for DF-HTA (4;7;8) but did not attempt a comprehensive characterization of this form of HTA. There was a recognition from this research group that this work, undertaken primarily for SMEs in the assessment of devices, was “a different animal” from use-focused HTA. For example, McAteer et al. used the term “supply side” HTA (Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8). However, we believe that this is the first comprehensive attempt to set out the features of HTA to inform developers. Our characterization is based on the extensive experience of the authors as well as the methodological and applied studies identified in our literature review.
The formal validation of our framework is limited by the, understandable, limited number of published examples of DF-HTA, especially commercial examples. There is little incentive for developers to publish HTA studies and the need for commercial confidentiality creates a disincentive. This means that the body of published literature is skewed toward work funded by a public body and/or supported by translational research bodies. A recent useful paper by Grutters et al. (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1) highlighted this bias. It summarized thirty-two assessments of thirty non-drug technologies undertaken by their academic group in the Netherlands. Of the thirty-two studies, thirty were designed to inform developers and all but two were unpublished. All the developers were small- or medium-sized enterprises. The features described by Grutters et al. (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1) supported our framework concerning the range of decisions to be informed and broad decision space. Timing of the assessments in this study ranged from idea screening, through concept development, pre-market and market access. Fifty percent of the technologies assessed were already available on the market so the timing is potentially a little later than we envisaged in our framework.
For analysts outside of large device or drug companies, new to this work, our framework provides a clear introduction to the features of DF-HTA and will guide their discussions with developers to ensure both parties are clear on the distinct nature of this work. It should also improve the transparency of any published DF-HTA if the features of each study are reported. For the wider academic community, we hope that our initial characterization of HTA to inform developers will provoke debate among practitioners about the nature of this work and the accuracy of our framework. Further research which would be of use include studies examining the features of DF-HTA in the commercial context and empirical studies applying our suggested framework. The different features of DF-HTA also necessarily impact on methods adopted. This has been explored for Early HTA in the academic literature (Reference Markiewicz, van Til and IJzerman13) but the boundaries of DF-HTA were not clearly established and this area warrants further exploration.
Health technology assessment (HTA) conducted to inform developers of health technologies is typically characterized in the academic literature as “early” health technology assessment. We prefer the label “development-focused HTA” (DF-HTA) as is it explicitly describes the purpose of the analysis, rather than alluding to just one, arguably not defining, characteristic.
DF-HTA has a number of distinct features when compared to HTA conducted to inform reimbursement and usage decisions (use-focused HTA). To conduct effective DF-HTA, it is important that analysts, who are often more familiar with use-focused HTA, are made aware of these differences. This is particularly important as DF-HTA analyses conducted or commissioned by commercial technology developers and in-house analyses undertaken in pharmaceutical and large medical device companies are typically not published due to a desire to maintain confidentiality and lack of incentive to publish (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1). Assessments of medical devices developed by small and medium enterprises (SMEs) or academic groups may be published if some public funding has been provided. Consequently, the studies which are published are likely to be a biased sample of the work undertaken and may not be particularly useful as a reference source for HTA analysts new to working with developers.
There is some useful methodological content in the published literature. For example, the Multidisciplinary Assessment of Technology Centre for Healthcare (MATCH) collaboration in the UK aimed to support companies in the UK healthcare technology sector to assess the value of medical devices from concept through to mature product. The collaboration extended the concept of iterative economic evaluation described in the late 1990s and 2000 (2;3) to develop methods and tools (e.g., Reference Cosh, Girling, Lilford, McAteer and Young4–Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8) for SMEs. The SMEs were often working in resource-constrained environments and had little in-house knowledge or experience of HTA (Reference Craven, Allsop, Morgan and Martin9). Other groups, particularly translational research bodies such as ProHTA (Reference Kolominsky-Rabas, Djanatliev, Wahlster, Gantner-Bär, Hofmann and German10) and the Center for Translational Molecular Medicine (Reference Steuten11), built upon the MATCH work and published further methodological and applied papers (e.g., Reference Ijzerman and Steuten12–Reference de Graaf, Postmus, Westerink and Buskens14). This form of HTA, initially described as “supply side” HTA by McAteer et al. of the MATCH collaboration (Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8), has more recently been described as “early HTA” (12;15).
We find the term “early HTA” as a term to describe DF-HTA somewhat vague and unhelpful. It might be taken to imply that methods and approaches for DF-HTA are essentially similar to those of the more commonly reported use-focused HTA, only undertaken at an earlier point in time. This problem is compounded by the fact that many early health economic modeling studies in the published literature take the normative structure of use-focused HTA and apply it at an earlier point in time. However, we argue that there are fundamental differences between development- and use-focused HTA that arise as a consequence of the differences in the target audience and the decisions that the analysis is intended to inform and that these are more important than the timing of the analysis.
In this paper, we aim to produce a characterization of DF-HTA that is useful to analysts new to working in this field. The framework is intended as an aide-memoire for analysts more familiar with use-focused HTA. We suggest that it is used at the outset of a project to prompt reflection on the nature of the assessment and to help structure discussions with developers. It can also be used at the reporting stage of a project, whether published or not, to ensure transparency. We hope that the suggested framework forms the starting point for a debate in the wider academic community about the nature of HTA undertaken to inform developers of health technologies.
To develop the framework of features of DF-HTA, we set out an initial framework based on our existing experience and a review of published literature. The literature review used a pearl-growing approach (Reference Claxton, Martin, Soares, Rice, Spackman and Hinde16) which aimed to identify examples of applied and methods papers where the intended audience was developers of medical technologies. We used Ijzerman and Steuten's 2011 review article (Reference Ijzerman and Steuten12) as the initial pearl. This approach involves reviewing the references and citations of the pearl for articles of interest, then reviewing the references and citations of the articles of interest until saturation is reached. The search was undertaken in October 2017 and refreshed in February 2019.
Features of Development-Focused HTA
Pietzsch and Paté-Cornell in 2008 (Reference Pietzsch and Paté-Cornell15) and Ijzerman and Steuten in 2011 (Reference Ijzerman and Steuten12) identified four features distinguishing early HTA from “Classical” or “Mainstream” HTA as they termed HTA undertaken to inform decisions taken at the market access stage. These features were: target audience; decisions to be informed; available evidence; and, timing. In addition to these four features, we identified a further four features in our literature review. These were: underlying user objective; decision space; business model; and resources for analysis. We then added a further two features of DF-HTA based on our experience: stance of analysis and burden of proof. The ten features are presented as a proposed framework in Table 1. These identified features are not separate and independent but intimately linked. For example, evidence is lacking because of the timing of the assessment. However, they are each worthy of explicit consideration.
Table 1. Features of DF-HTA
HTA, health technology assessment.
Target Audience
The target audience for DF-HTA includes both the developers of technologies and the sponsors or funders of the development. These may include both commercial and academic institutions as well as private and public sectors funders (Reference Pietzsch and Paté-Cornell15). We will hereafter use the general term “developers.” The target audience of developers is, in our view, the defining feature of this form of HTA. DF-HTA differs from other forms of HTA because of the requirements of its target audience. In published studies, the target audience is often not explicitly defined (e.g., Reference Hjelmgren, Ghatnekar, Reimer, Grabowski, Lindvall and Persson17–Reference Vilsbøll, Mouritsen, Jensen, Bødker, Holst and Pennisi19). In some cases, the analysts appear to adopt the perspective of a payer even when the HTA is undertaken to inform the developer. For example, Latimer et al. (Reference Latimer, Dixon, McDermott, McCarthy, Tindale and Heron20) undertook an economic evaluation to inform developers about the feasibility of designing a collar for use by patients with motor neurone disease which would be cost-effective from the perspective of UK NHS. Such an analysis fails to explicitly recognize that the technology might be marketable in multiple markets that apply differing criteria to determine reimbursement.
Underlying User Objective
The primary objective for a commercial sector developer or investor is to maximize long-term financial return on investment (12;21–23). Other social objectives or motivations are typically subservient to this objective. The primary objective of public sector developers (e.g., academic developers funded by public bodies) is to maximize the societal return on investment. Societal return includes consideration of direct financial returns on development, industrial growth or employment, and improvements in societal health [e.g., Innovate UK's funding streams (24)].
To maximize long-term financial return on investment, developers and investors need to consider the measures of value for money that payers use in their coverage decisions. Thus, the underlying objective of the payer is relevant for developers to inform pre-clinical, preliminary market assessments, and first estimations of pricing and reimbursement scenarios. Although explicit thresholds, such as the £20,000 to £30,000 per quality-adjusted life-years in the UK (25), are often used in DF-HTA to make a first estimate of the maximum price achievable for a technology in order for it to be considered cost-effective (7;8;26) it is important for DF-HTA analysts to recognize that a range of approaches are used by different payers.
The underlying decision rules used by decision makers should reflect their objectives. Analysts undertaking DF-HTA should acknowledge that commercial developers and investors will use, either implicitly or explicitly, a decision rule based on the expected net present value of an investment. This means that they will be interested in the expected revenues to be generated across relevant markets and the expected costs associated with delivering these sales as well as the timescales over which the revenues and costs occur. In principle, the net present value of these potential inflows and outflows should be calculated using a discount rate which takes into account the company's cost of borrowing reflecting the perceived risk of the project. In practice, crude measures of the opportunity costs of a particular investment are likely to be used.
Public funders and non-commercial developers may base a decision to continue the development on a formal net value of information analysis based on the acceptable cost-effectiveness threshold (which itself should represent the opportunity cost of healthcare expenditure). More informal analyses may simply try to estimate the likelihood or potential that a technology will be regarded as cost-effective. In this situation, the decision rule used is to continue the development if the technology is likely to meet the appropriate thresholds for cost-effectiveness in the relevant jurisdiction. Commercial developers and investors may also be interested in the outcome of this analysis as it would provide some indication of the likelihood of sales in the relevant jurisdiction and provide guidance as to acceptable pricing. It should be recognized that explicit thresholds are not used by all payers, they vary between jurisdictions, they are not the only determinant of reimbursement, and they are subject to change. Commercial developers and investors may also take into account other aspects of value not typically included in formal cost-effectiveness analyses, such as patient convenience or comfort and the value of knowing a diagnosis for patients and their families, if these are likely to influence usage (18;27).
Decisions HTA Designed to Inform
DF-HTA potentially informs a wide range of decisions and considerations including: preliminary market assessment; estimation of pricing; review of reimbursement scenarios; individual go/no go decisions; technology design; evidence generation strategy including study design; and research and development portfolio prioritization (15;28). As DF-HTA is undertaken before the development process concludes, developers can respond to the assessment by changing the design of the technology, its target indication(s) and position in the clinical pathway (Reference Pietzsch and Paté-Cornell15). The assessment process itself may highlight gaps in the evidence for the new technology which can drive the evidence generation strategy at the next phase of development. This can also facilitate discussions with regulators or reimbursement agencies that increasingly offer to engage with developers during the development process. If assessment is undertaken simultaneously for a number of technologies, the results can be used to identify the most promising technologies facilitating the prioritization of research effort and expenditure. For example, de Graaf et al. (Reference de Graaf, Postmus, Westerink and Buskens14) assessed the potential of biomarker tests in four roles in the prevention of type-2 diabetes mellitus to prioritize research effort and expenditure within a translational research organization.
Decision Space
By decision space, we mean the range of different ways and places in which a technology may be used, for example, clinical indication, target population, and placement in the treatment pathway. In DF-HTA, the decision space is often wide and poorly defined. As DF-HTA is generally undertaken prior to licensing, the potential indications and positions in the clinical pathway are not yet constrained by licensing restrictions and multiple options may need to be assessed (Reference Annemans, Genesté and Jolain3). Other aspects of decision space include multiple versions of the technology (including optimization of test characteristics for diagnostics) (Reference Buisman, Rutten-van Mölken, Postmus, Luime, Uyl-de Groot and Redekop23), patient populations (11;21), jurisdictions, comparators, dosages, modes of delivery, pricing structures (Reference Annemans, Genesté and Jolain3), and diffusion scenarios (Reference Rogowski, John, Ijzerman and Scheffler27). Furthermore, these may vary across different potential markets.
Available Evidence
In DF-HTA, evidence specific to the technology is typically scarce early in the development process. As direct evidence of clinical effectiveness is lacking there is more reliance on elicited expert opinion (4;21), evidence relating to comparator technologies (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton21), bench or animal studies, previous generations of a technology (Reference Pietzsch and Paté-Cornell15), and extrapolations from pharmacodynamics models (Reference Annemans, Genesté and Jolain3). Where direct clinical evidence is available, studies are often small so that uncertainty around any estimates is high. Methods of expert elicitation have been developed to improve the reliability of experts' estimates of plausible ranges. Evidence may also be required about usability or the impact of a technology on clinical pathways (Reference Abel, Shinkins, Smith, Sutton, Sagoo and Uchegbu29). Qualitative methods (30;31) and multi-criteria decision analysis (Reference Hummel, Van Rossum, Verkerke and Rakhorst32) have been used to address this need. Shortage of evidence is not unique to DF-HTA, as uncertainty is inherent in all HTA. However, the shortage is likely to be more pronounced earlier in a development process.
Timing
We see DF-HTA as an ongoing activity facilitating a continuous discussion around the technology development process rather than a discrete event with a specific output. The majority of DF-HTA will be undertaken before a technology is approved by a regulatory body. The starting point for the DF-HTA may be the identification of a clinical need preceding the product development process (Reference Yock, Zenios, Makower, Brinton, Kumar and Watkins33). In this case, the DF-HTA would assess the potential for the technology proposed. An example of this approach is provided by Brandes et al. (Reference Brandes, Sinner, Kääb and Rogowski34), who assessed a hypothetical vascular closure device and found only a single sub-group where the technology had potential. Alternatively, the starting point for DF-HTA may be the evaluation of a technology already in development. Kluytmans et al. (Reference Kluytmans, Tummers, van der Wilt and Grutters31) evaluated a surgical device at prototype stage and found that there was little potential for the device in meniscus surgery, which was the developers' suggested indication. DF-HTA is particularly suited to an iterative approach with discussions with developers continuing alongside the development process and analysis undertaken prior to significant investments, such as Phase II or Phase III trials for pharmaceuticals (5;6). Vallejo-Torres et al. (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton21) presented an iterative economic evaluation of absorbable pins for hallux valgus at three different stages of development. The authors used retrospective data for this analysis to recreate the dynamic process of DF-HTA occurring in real-time alongside the development process. It should also be noted that use-focused HTA may also use an iterative approach (rather than the discrete event with a specific output described above), as products are arriving to market with greater levels of uncertainty.
Business Model
In this context, the term “business model” broadly refers to how a technology and the customer are brought together, which determines how the revenue stream is generated and what barriers there may be to entry (Reference Rogowski, John, Ijzerman and Scheffler27). In DF-HTA, the business model may not be fixed. Developers have the option to offer their technology (subject to local regulatory constraints) wherever the potential is greatest and to target patients and/or clinicians directly or to sell via national health services. For example, van Nimwegen et al. (Reference van Nimwegen, Lilford, van der Wilt and Grutters18) used parents' willingness to pay for a diagnosis to calculate “headroom” (valuing an estimated extension in life and/or improvement in quality of life at a given threshold value with an adjustment for the cost impact of the technology) rather than an explicit threshold for reimbursement as it was felt that the technology would be best suited to the private payer market. The business model adopted by the commercial developer or investor may differ across jurisdictions. Non-commercial developers may also need to consider commercial means of bringing their technology to market, as established biotech companies maybe best-placed to maximize the technology's potential.
Resources for Analysis
In the early stages of development, in large companies, there may be a set of candidate technologies which could potentially be assessed using DF-HTA. As many of these potential technologies will fail (3;5) resource-intensive approaches to HTA themselves may not have a positive expected net present value. DF-HTA must compete for scarce resources, potentially displacing aspects of the research and development process. In addition, many medical devices, including tests, are developed by small- and medium-sized enterprises and may be the sole product of that company (Reference Craven, Allsop, Morgan and Martin9). Such companies may have limited HTA experience and resources. This means that DF-HTA must deliver value within significant resource constraints. At the earliest stages of development, it is suggested that effort is focused on articulating and quantifying a value proposition (5;27). This could potentially be done using qualitative interaction with clinicians and users (30;31) and simple quantitative methods such as headroom analysis (Reference Chapman7). This prima facie case can then be developed further as the development progresses when more resources may be available (5;23;27).
Stance of Analysis
By stance of analysis, we mean the mindset adopted by the analyst in undertaking the assessment. We believe that the adoption of a positive rather than a normative economic stance of analysis is one of the fundamental features of DF-HTA, which has not previously been widely discussed. DF-HTA for commercial developers adopts a positive stance, as no value judgments are required (Reference Culyer35) and the analysis is focused on the maximization of the developers' return on investment. For example, Hummel et al. (Reference Hummel, Boomkamp, Steuten, Verkerke and Ijzerman36) mentioned that the aim of their analysis was to “support the future development” of the technology. Similarly, Kluytmans et al. (Reference Kluytmans, Tummers, van der Wilt and Grutters31) commented that much early HTA “has a strong technology-focused or supply-driven character.” We concur with this statement as, in our experience, developers start with the technology and part of the role of DF-HTA is to find a place where it can be successful. In this sense, DF-HTA has the character of a formative assessment, that is, an assessment to further the development. By way of contrast, use-focused HTA has the character of a summative assessment against a pre-determined set of criteria. Use-focused HTA adopts a normative stance; it involves judgments about what is good for society (Reference Culyer35).
Burden of Proof
There are no guidelines about either methods to be adopted or the acceptable level of evidence required for DF-HTA, nor would such guidelines be appropriate. The process of DF-HTA is iterative; initial stages use whatever evidence is available and methods deemed appropriate by the analyst. The output from the HTA process informs the discussion between the developer and the analyst and takes any limitations in evidence and methods of assessment into account. For use-focused HTA, in many jurisdictions, there are clear guidelines as to what level and form of evidence the reimbursement agency or payer deems acceptable as well as how the assessment should be undertaken. For example, the National Institute for Health and Care Excellence, the reimbursement agency for England and Wales, prefers the evidence of health effects to come from randomized-controlled trials directly comparing the intervention with one or more relevant comparators and has a comprehensive guide to methods (25).
How the Framework may be Used
We suggest that our framework of features of DF-HTA is used as an aide-memoire at the planning stage of a project, in initial discussions with developers. This would help to clarify essential features of the analysis in the mind of the analyst and ensure transparency between the developers and the analyst. Certain features of the framework may encourage discussions about features which would be unlikely to be discussed otherwise, such as the developers' underlying objective. It may also encourage a consideration of the wider decision space or alternative business models. Additionally, the framework could be used as a checklist for reporting to developers or in a published article to ensure that the characteristics of the analysis are transparent. Table 2 shows a summary of questions for consideration or discussion.
Table 2. Questions for consideration in DF-HTA
DF-HTA, development-focused HTA, HTA, health technology assessment.
Discussion
We set out to provide a characterization of HTA undertaken to inform developers. We described ten features of DF-HTA in a framework to be used as an aide-memoire for analysts new to this work and as a checklist for reporting. Four of the features (target audience, decisions to inform, available evidence, and timing) had been included in previous frameworks distinguishing early and mainstream HTA (Reference Ijzerman and Steuten12) or classical HTA (Reference Pietzsch and Paté-Cornell15). The remaining features (underlying user objective, decision space, business model, resources available for analysis, stance of analysis, and burden of proof) were identified by the authors, informed by our experience and the methodological and applied papers identified in our review.
Although previous authors have gone some way toward characterizing DF-HTA (as part of “Early HTA”) (12;13;15), it was often conflated with other activities where evidence was scarce such as horizon-scanning (Reference Ijzerman and Steuten12) and the assessment of process or innovation from the perspective of the health service provider (13;37). Although these related activities may share some features with DF-HTA such as the timing and the lack of evidence, they differ significantly in important aspects of the work. In particular, the target audience for the work is healthcare decision makers and the stance of analysis may be normative in nature. Authors associated with the MATCH collaboration in the UK set out a methodology for DF-HTA (4;7;8) but did not attempt a comprehensive characterization of this form of HTA. There was a recognition from this research group that this work, undertaken primarily for SMEs in the assessment of devices, was “a different animal” from use-focused HTA. For example, McAteer et al. used the term “supply side” HTA (Reference McAteer, Cosh, Freeman, Pandit, Wood and Lilford8). However, we believe that this is the first comprehensive attempt to set out the features of HTA to inform developers. Our characterization is based on the extensive experience of the authors as well as the methodological and applied studies identified in our literature review.
The formal validation of our framework is limited by the, understandable, limited number of published examples of DF-HTA, especially commercial examples. There is little incentive for developers to publish HTA studies and the need for commercial confidentiality creates a disincentive. This means that the body of published literature is skewed toward work funded by a public body and/or supported by translational research bodies. A recent useful paper by Grutters et al. (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1) highlighted this bias. It summarized thirty-two assessments of thirty non-drug technologies undertaken by their academic group in the Netherlands. Of the thirty-two studies, thirty were designed to inform developers and all but two were unpublished. All the developers were small- or medium-sized enterprises. The features described by Grutters et al. (Reference Grutters, Govers, Nijboer, Tummers, Van Der Wilt and Rovers1) supported our framework concerning the range of decisions to be informed and broad decision space. Timing of the assessments in this study ranged from idea screening, through concept development, pre-market and market access. Fifty percent of the technologies assessed were already available on the market so the timing is potentially a little later than we envisaged in our framework.
For analysts outside of large device or drug companies, new to this work, our framework provides a clear introduction to the features of DF-HTA and will guide their discussions with developers to ensure both parties are clear on the distinct nature of this work. It should also improve the transparency of any published DF-HTA if the features of each study are reported. For the wider academic community, we hope that our initial characterization of HTA to inform developers will provoke debate among practitioners about the nature of this work and the accuracy of our framework. Further research which would be of use include studies examining the features of DF-HTA in the commercial context and empirical studies applying our suggested framework. The different features of DF-HTA also necessarily impact on methods adopted. This has been explored for Early HTA in the academic literature (Reference Markiewicz, van Til and IJzerman13) but the boundaries of DF-HTA were not clearly established and this area warrants further exploration.
Conflict of interest
Briggs reports grants from MRC during this study. Bouttell and Hawkins have nothing to disclose.