Each year a huge number of medical devices are being developed, but only a few make it to the market (Reference Bartelmes, Neumann, Lühmann, Schönermark and Hagen1). The development process of medical devices is a costly and uncertain undertaking. Failed development does not only result in lack in economic return for the company, but also in high costs without healthcare improvements for society (Reference Bartelmes, Neumann, Lühmann, Schönermark and Hagen1–Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3). There are multiple reasons for failed device development, but one important factor is the late evaluation of the potential of the device in healthcare practice, usually only after the prototype design is finalized. Various authors suggested that assessment of medical devices early in the development process, at the stage where it is still possible to curtail the diffusion or influence their development in simple and inexpensive manner, may be beneficial (Reference Bartelmes, Neumann, Lühmann, Schönermark and Hagen1;Reference Goodman and Ahn4–Reference Ibargoyen-Roteta, Gutierrez-Ibarluzea, Asua, Benguria-Arrate and Galnares-Cordero7).
Based on the health technology assessment (HTA) definition of the International Network of Agencies for Health Technology Assessment “early assessment of medical devices” can be defined as the early examination of the medical, economic, social, and ethical implications of the medical device to determine the potential for incremental value in healthcare (8). It starts from initial idea generation up to stage I of clinical trials (Figure 1) (Reference Cosh, Girling, Lilford, McAteer and Young9–Reference Van der Wetering, Steuten, von Birgelen, Adang and IJzerman12). At each of the stages, different qualitative and quantitative assessment methods can be used, to provide information that is of interest in that stage to feed the decision-making process of the responsible stakeholders (Reference IJzerman and Steuten10).
Figure 1. A simplified flow-chart of stages in medical product development based on IJzerman and Steuten, 2011 [10].
At present, most of the decisions made early in development seem to be taken quickly and in the absence of good quality evidence, although those decisions can have a long-term impact on device design (Reference Robertson, Watkins and Reily21). Early phases of development are characterized by manufacturers enthusiasm, competition, and desire to pioneer, which can result in false judgment based information which relies on insufficient information (Reference Ballini, Minozzi, Negro, Pirini and Grilli13). Early assessment of the medical device in the healthcare context could help to support and guide decisions with as much evidence or motivated assumptions as possible (Reference Girling, Young, Brown and Lilford14).
The aim of early assessment is to reduce the failure rate at each stage of the development process, while enhancing the efficiency of R&D and of limited resources use, through prioritization of the innovations most likely to succeed among others. It may also be used to support reimbursement claims by providing quantitative input for developing risk-sharing agreements (Reference Lim, O'Reilly and Tarride2;Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3;Reference Hilgerink, Hummel, Manohar, Vaartjes and IJzerman11;Reference Ahn, Zwikael and Bednarek15;Reference Hartz and John16).
Several studies have indicated the importance of on-going assessment as an integral part of the medical devices development process (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3;Reference Girling, Young, Brown and Lilford14;Reference Sculpher, Drummond and Buxton17;Reference Fenwick, Palmer and Claxton18), but there is still lack of general understanding on which methods should be used at the different stages, which data should be gathered to inform early decision making, as well as how to use the results to inform stakeholders. The overall aim of this study is to describe the current state of the art in early assessment and to identify assessment methods that help to inform decisions during the development stage of medical devices, so that medical devices are less likely to be market failures and more likely to be approved by regulatory and reimbursement agencies. This implicit aim is important and has not been previously answered.
METHODS
Searching for Relevant Studies
A systematic literature review was performed to identify studies reporting on early assessment to help inform the early development of medical devices. The first objective was to select theoretical and application papers reporting on early assessments of medical devices. The second objective was to identify the assessment methods in use and map their use throughout the aims and stages of early development.
A systematic search strategy was conducted in September 2013 using (i) computerized databases (PubMed, Science Direct, and Scopus) and (ii) reference search of included articles. Search strategies were built based on the keywords, such as technolog*, approval, biomedical, design, early HTA, equipment, assessment, medical development, model*, device, valu*, healthcare, R&D, strategic plan*, innovat*, cost*, health, project management, decision mak*, and medical subject headings (MeSH): Technology assessment, biomedical; Biomedical technology; Technology, high cost; Device approval; Equipment design; Technology transfer. Supplementary Table 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000026, provides a detailed overview of the search strategies and their results. Furthermore, reference lists of included papers were hand searched.
Selection Criteria for All Studies
The selection was restricted to articles in English, involving human subjects and published after 1996, as the growing interest in methods that more specifically inform decisions in earlier stages of product development is a fairly recent trend. Only full journal articles and papers with ISBN and ISSN numbers were included in the review. The review of the articles was accomplished in two consecutive screenings. In the first screening, the titles and abstracts were reviewed for relevance by two authors (K.M. and J.v.T.) according to the following inclusion criteria: (i) the articles written within the healthcare context; and (ii) articles reporting on theory or practice of assessment of a medical technology. Relevant articles were obtained as a full text and assessed against the selection criteria (K.M. and J.v.T.). Disagreements were resolved by discussion or referred to a third author (M.I.J.). Articles eligible for the review were chosen after the careful reading of the full article. In this stages, articles were excluded if they did not report on early assessment (as defined below) of medical devices.
Based on the definition provided by the U.S. Food and Drug Administration a “medical device” was defined as an instrument, apparatus, implant, in vitro reagent, or similar or related article that is used to diagnose, prevent, or treat disease or other conditions, and does not achieve its purposes through chemical action within or on the body. “Early assessment of medical device” was defined as the assessment of the value of medical device under development at the time when investments and design decisions have to be made with high uncertainties about future prospects, up to stage I of clinical trials (when the design is mostly finalized and the device is not yet implemented). Because of the broad nature of the study aims and the wide variety of studies applying different methods that were included, no quality instrument was available for the authors to use.
Categorization of the Papers
All articles selected for the review were categorized as “theoretical papers,” which aimed at building a framework for early assessment (including systematic reviews of existing literature) or “application papers,” which are case studies of early assessment, or illustrations of theory using examples.
Data Extraction and Synthesis
From each paper, the study objectives and assessment methods were identified and grouped based on common aims. The study objectives were next classified for specific target audience they aimed to inform, and the assessment methods were classified based on the early stage of medical device development they were used at. Additionally, the methods used in the application papers were identified and classified into either qualitative or quantitative. The following outcomes were extracted:
Main target audience: Decision makers on coverage and reimbursement; policy makers; manufacturers; varied. The main target audience was determined based on the early assessment decision support system presented by Pietzsch and Paté-Cornell (Reference Pietzsch and Paté-Cornell19).
Device development stage: Basic research on mechanisms; targeting for specific product; proof of concept; prototype product development; first clinical trials; not specified. The development stages were determined based on the simplified framework presented in Figure 1.
Study objectives and early assessment methods: Not pre-specified but categorized on the basis of the data obtained.
Categorization of the outcome variables was based upon the agreement between the authors.
RESULTS
Literature Search Strategy
Figure 2 presents a flow chart of the literature selection procedure. The systematic literature search yielded 1,961 hits. Eighty-three articles immediately met the inclusion criteria and were selected from the search strategy. Another thirty articles were selected based on screening of the references in the selected articles. Eighty-two of the 113 selected studies were application papers, and thirty-one studies were theoretical papers.
Figure 2. Flow chart: selection of the literature.
Early Assessment Objectives in Theory and in Practice
Table 1 presents the study objectives of selected articles, which are categorized according to their main target audience, and subdivided to theoretical and applications papers.
Table 1. Study Objectives in Selected Articles According to Addressed Main Target Audience
T, theoretical papers (incl. systematic reviews); P, application papers [incl. theoretical papers with example].
From the analysis (Table 1), it became apparent that the main target audience for early assessment are the manufacturers. Most application papers reported on the potential applications or improvement directions for a medical devices, in some cases in a specific disease context (thirty articles). Although it was decided that these papers mainly targeted manufacturers, who scan the emerging trends and developments in the targeted disease area for new insights to develop strategies for anticipating future developments (Reference Clopton and Spelman88–Reference Baumgart117), they could as well be addressed to the policy makers, who based on them decide on the societal funds. The analysis also revealed that there is a focus on developing a framework for early assessment of medical device (twenty articles). Some frameworks propose to support decision-making processes in medical device development through analytical decision support techniques (Reference Cosh, Girling, Lilford, McAteer and Young9;Reference IJzerman and Steuten10;Reference Ballini, Minozzi, Negro, Pirini and Grilli13;Reference Fenwick, Palmer and Claxton18;Reference Retèl, Hummel and van Harten62;Reference Robinson, Huang, Guo and Porter66;Reference Robinson and Propp70;Reference Rogowski, Hartz and John72;Reference Stevens, Milne, Lilford and Gabbay73), while others address the specific demands of the early development context (Reference Spiegelhalter, Myles, Jones and Abrams74–Reference Ladabaum, Brill and Sonnenberg76). A third important aim in early assessment is to include end-user perspectives in further development of a medical device (eighteen articles). These studies aim to convince manufacturers of the value of end-user perspective on development (Reference De Rouck, Jacobs and Leys49–Reference LeRouge, Ma, Sneha and Tolle52;Reference Shah, Robinson and AlShawi55;Reference Hardisty, Peirce and Preece59), mainly through giving practical examples. Five studies focus on the theoretical development of this study area (Reference Yen and Bakken44;Reference Gallego, Bridges, Flynn, Blauvelt and Niessen45;Reference Money, Barnett and Kuljis47;Reference Shah and Robinson48;Reference Shah and Robinson60).
Systematic Analysis of Early Assessment Methods in Use
Table 2 presents different qualitative and quantitative methods used in the application papers at different stages of early development of medical devices grouped according to their aims. Supplementary Table 2, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000026, presents definitions and objectives of those methods.
Table 2. Quantitative and Qualitative Methods Used in the Early Assessment of Medical Devices According to the Stage of Development and Their Aims
Note. Qualitative methods used in the early assessment of medical devices: Literature review/analysis (e.g. archives, documents) [1]; Peer review [2]; User profiles building [3]; Focus groups [4]; Interviews (e.g. experts) [5]; Informal discussions [6]; Qualitative weighing of relevant factors [7]; Use cases writing [8]; Key informant interviews [9]; Strategic planning methods: PEST, SWOT [10]; Soft Systems Methodology [11]; Expert panels/elicitation [12]; Technology profiling (uncertainty profile and evidence profile) [13]; Workshops [14]; Surveys [15]; R&D portfolio management [16]; Brainstorming sessions [17]; Users-producers seminars [18]; Usability tests [19]; Users feedbacks [20]; Clinical trials [21]; Choice-based conjoint analysis (Discrete choice modelling) [22]; Horizon scanning [23]; Preliminary market research [24]; Bench studies [25].
Quantitative methods used in the early assessment of medical devices: Headroom analysis [I]; Cost-effectiveness analysis (CEA) [II]; Probabilistic sensitivity analysis (PSA) [III]; Potential years of life lost (PYLL) [IV]; Cost-benefit analysis (CBA) [V]; Cost-utility analysis (CUA) [VI], Opportunity costs (used as indicators to which relative weights are assigned) [VII]; Roadmapping process (Multi-Path Mapping) [VIII]; Scenarios building [IX]; Return on investment [X]; Technological forecasting based on epidemiological data [XI]; Rudimental analysis of costs [XII]; MultiCriteria Decision Analysis (Analytic Hierarchy Process) [XIII]; VOI Expected Value of Perfect Information (EVPI) [XIV]; Bayesian modelling/statistics (data pooling, random effects analysis) [XV]; Probabilistic Risk Analysis [XVI]; Real options analysis [XVII]; Best-worst scaling (BWS) [XVIII]; Decision tree analysis [XIX].
Most of the application papers do not specify the stage of device development of the device or they reported that the early assessment took place in the prototype product development phase. There is great diversity in the methods used in early assessment and in their goals. Early assessment comprises a strategic analysis (including stakeholders analysis) of the medical context and the competition, evaluation of the economic impact of medical devices and early assessment of clinical effectiveness of the medical devices under development, all with the aim to reduce uncertainty in the developmental stage of a medical device. Qualitative and quantitative research methods are about equally applied in the different stages of medical device development, but do differ based on aims.
Main Objectives of Early Assessment
Strategic Considerations
A large focus in early assessment is on strategic considerations. In assessing strategic issues that could influence development, two methods of study can be distinguished: literature review and stakeholder involvement. Desk search analysis is usually performed to analyze the market/knowledge gaps and potential applications for medical devices under development, through literature review/analysis (Reference Cosh, Girling, Lilford, McAteer and Young9;Reference Ballini, Minozzi, Negro, Pirini and Grilli13;Reference Chávez-Santiago, Balasingham and Bergsland29;Reference De Rouck, Jacobs and Leys49;Reference Hardisty, Peirce and Preece59), SWOT (Strengths, Weaknesses, Opportunities, and Threats analysis), or PEST (Political, Economic, Social, and Technological analysis) analysis (Reference Cosh, Girling, Lilford, McAteer and Young9;Reference Ahn, Zwikael and Bednarek15) and/or horizon scanning (Reference Gallego, Bridges, Flynn, Blauvelt and Niessen45;Reference Douw, Vondeling and Oortwijn78;Reference Douw, Vondeling, Sørensen, Jørgensen and Sigmund82;Reference Paradise, Diliberto, Tisdale and Kokkoli119). The aims of stakeholder involvement, stakeholder analysis (Reference De Rouck, Jacobs and Leys49–Reference Sintonen and Immonen57) is thought to increase understanding of needs and wants of policy makers and end users to tailor their device to the health context (Reference Bridgelal Ram, Grocott and Weir58;Reference Backhouse, Wonder and Hornby120). Main methods used for stakeholder involvement are focus groups, interviews, expert panels, workshops, or surveys (Reference Ahn, Zwikael and Bednarek15;Reference Brodtkorb33;Reference LeRouge, Ma, Sneha and Tolle52;Reference Coughlin, Pope and Leedle54;Reference Sintonen and Immonen57;Reference Hardisty, Peirce and Preece59;Reference De Rouck, Jacobs and Leys121). The great majority of studies were qualitative in nature, although in some cases ranking or rating of factors took place. One of the more often used quantitative methods was a Bayesian modeling/statistics, which is based on modeling of the evidence about the true state of the world expressed in terms of degrees of belief, and scenarios building—a creative method for trend extrapolation and envision of alternative paths into the future.
Economic Evaluations
Economic evaluation is in its nature a quantitative methods. Cost-effectiveness analysis (CEA) with subsequent probabilistic sensitivity analysis (PSA) was performed starting with the proof of concept stage of the development. An interesting new technique used in early assessment is the Headroom Method, which uses broader estimates of potential by determining the maximum reimbursable price of the new device, and is especially tailored to the early assessment needs of medical devices. There is also a focus on studying the impact of different types of uncertainty in development on decision making, for example, by eliciting the willingness to pay of decision makers for additional information to avoid uncertainty, such as value of information (VOI).
Clinical Considerations
Clinical assessment methods in early development are, next to the classical clinical trials, those performed in a controlled laboratory setting such as bench studies, where the performance of the medical device is compared with a gold standard, or clinical practice. Although findings of this review show that clinical effectiveness assessment of the medical devices starts late, at the prototype product development phase, in practice clinical research is also a part of methods assigned to other aims, such as CEA, or cost-benefit analysis (CBA).
DISCUSSION
The overall aim of this study was to describe the current state of the art in early assessment and to identify assessment methods that help to inform the early development of medical devices so that medical devices are less likely to be market failures and more likely to be approved by regulatory and reimbursement agencies. This study yielded 113 papers on early assessment. As can be expected, most studies were aimed at informing manufacturers of medical devices on the potential of their device. Kazanjian and Green (Reference Kazanjian and Green71) recognized that manufacturers usually have a quite restricted viewpoint during development, which mainly focuses on demonstrating proof of concept of the technology. Early assessment can partly overcome this by evaluating a device in its clinical setting, within the current healthcare market and with respect to its potential for bringing benefit to the company and society (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3). Although there is also lack of any evidence on how effective the identified assessment methods are and what is their actual influence on the decision-making process, different studies stress the need for manufacturers to systematically acquire information to feed their decision-making process in early development (Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3;Reference Cosh, Girling, Lilford, McAteer and Young9;Reference IJzerman and Steuten10;Reference Hartz and John39;Reference Goeree, Levin and Chandra122;Reference Abalos, Carroli and Mackey123). The question on how the effectiveness of this early evidence could possibly be measured is open for further research.
Analysis of study objectives within early assessment of medical devices showed that studies into the strategic issues within the healthcare context and studies on the economic impact of medical devices are well represented. Exploration of the potential of a medical device from a strategic perspective is often used in business plans, to identify the main barriers for successful development in all stages of development. The focus on demonstrating economic impact from a societal perspective in early assessment is probably explained by the current paradigm in traditional health technology assessment, in which demonstrating cost-effectiveness of drugs is an important hurdle to reimbursement (Reference Van der Wetering, Steuten, von Birgelen, Adang and IJzerman12;Reference Van Til, Renzenbrink, Groothuis and IJzerman31;Reference Dougherty32;Reference McAteer, Cosh and Freeman34–Reference Pertile36).
Although individual methods might be well developed, there is no agreed-upon theoretical framework for early assessment. The interest in early assessment from a scientific perspective has resulted in the proposal of multiple, sometimes overlapping frameworks (Reference Cosh, Girling, Lilford, McAteer and Young9;Reference Ballini, Minozzi, Negro, Pirini and Grilli13;Reference Pietzsch and Paté-Cornell19;Reference Hummel, Boomkamp, Steuten, Verkerke and IJzerman63–Reference Robinson, Huang, Guo and Porter66;Reference Robinson and Propp70;Reference Kazanjian and Green71;Reference Tal and Hakak77–Reference Wild and Langer81;Reference Berry83;Reference O'Malley and Jordan84). The lack of uniformity to the process is also related to the dynamic nature of the device development process which requires flexibility in the assessment process (Reference Cosh, Girling, Lilford, McAteer and Young9;Reference Sculpher, Drummond and Buxton17;Reference Santler22;Reference Money, Barnett and Kuljis47–Reference LeRouge, Ma, Sneha and Tolle52). Medical devices are changing rapidly during their life cycle due to incremental product improvement, and they constitute moving targets for assessment (Reference Goodman and Ahn4;Reference Wild and Langer81). However, until a more unified theory behind the practice is developed and tested, the benefits of early assessment are difficult to evaluate. At present, there is no external motivator for manufacturers to perform early assessment and its implementation depends on demonstrating value in practice (Reference Pietzsch and Paté-Cornell19;Reference Hartz and John39;Reference Brown, Smale, Verma and Momandwall80).
One of the biggest challenges in early assessment is the way to handle uncertainty in interpreting the results (Reference IJzerman and Steuten10;Reference Girling, Young, Brown and Lilford14;Reference Bojke, Claxton, Sculpher and Palmer124). High uncertainty is inherent to the early development stage. If the uncertainty is not handled well it might cause misleading results in demonstrating future clinical and economic benefits, increasing the risk of making “wrong” decisions (Reference Lim, O'Reilly and Tarride2;Reference Vallejo-Torres, Steuten, Parkinson, Girling and Buxton3;Reference Cosh, Girling, Lilford, McAteer and Young9;Reference Wild and Langer81). However, the presence of uncertainty in the input parameters for early assessment should not result in refraining from analysis. Rather than trying to make decisions in the absence of evidence, one should attempt to estimate the influence of uncertainty and quantify or qualify its influence on decisions to be made in further development.
One important limitation in this study might be a publication bias. Because of the competitive nature of the medical device development process, manufacturers shield their information from others. It is unlikely that a manufacturer would allow publication of early assessment results before the device has reached the market, or has failed. Another limitation might be the search sensitivity, for example, due to the incorrect choice of the keywords or construction of search strategies.
CONCLUSIONS
The main target audience for early assessment are the manufacturers. Most application papers aimed at reporting on the potential applications or improvement directions for a medical device(s), development of a framework for early assessment of medical device, or stakeholders perspective inclusion in further development of a medical device. In most of the cases, application papers did not specify the stage of device development or they reported that the early assessment took place in the prototype product development phase. There is great diversity in the methods used in early assessment. Qualitative and quantitative research methods were about equally applied in the different stages of medical device development, but they differed based on aims. Early assessment includes a strategic analysis (with stakeholders analysis) of the medical context, evaluation of the economic impact and early assessment of clinical effectiveness of the medical devices under development. All the methods identified aim to reduce uncertainty in the developmental stage of a medical device. To inform strategic considerations, literature review and methods focused on stakeholder involvement (e.g., focus groups, interviews) were used frequently. CEA together with the Headroom method were often used as an economic evaluation methods, while clinical effectiveness of new devices was measured through clinical trials and bench studies.
Early assessment of medical devices under development holds the promise for more informed decisions, that could improve the pace and the efficiency of the development and guarantee successful implementation in the future. However, there is no well-developed framework for early assessment, which makes evaluation of its value difficult. For early assessment to become a practical tool to support manufacturers in medical device development, some basic classification and harmonization of methods are necessary.
SUPPLEMENTARY MATERIAL
Supplementary Tables 1 and 2: http://dx.doi.org/10.1017/S0266462314000026
CONTACT INFORMATION
Katarzyna Markiewicz, MSc (k.markiewicz@utwente.nl), Department of Health Technology and Services Research, University of Twente, Drienerloaan 5, 7500AE Enschede, The Netherlands
Janine A. van Til, PhD, Assistant Professor, Department of Health Technology and Services Research, University of Twente, Drienerloaan 5, 7500AE Enschede, The Netherlands
Maarten J. IJzerman, PhD, Professor, Department of Health Technology and Services Research, University of Twente, Drienerloaan 5, 7500AE Enschede, The Netherlands
CONFLICTS OF INTEREST
K. Markiewicz reports no conflicts of interest. J. van Til reports a grant to institution. M. IJzerman reports a grant to the author and institution from ISPOR Board of Directors, consultancy to the author and institution from Panaxea BV, employment money to the author from University of Twente, expert testimony to the institution from Panaxea BV, honoraria to the institution from Panaxea BV, and travel/accommodations expenses covered or reimbursed to the institution from University of Twente.
