In the area of biotechnological and genetic research, new findings are frequently published, leading to the development of medical innovations, public demand for fast implementation, and to coverage decisions in several cases. Since the 1970s, a health technology assessment (HTA) is used to support decisions of whether or not to implement these new technologies into clinical practice (Reference Berg, van and Klazinga2;Reference Leys9).
Technology assessment originated in the business and public policy arenas before it was adopted as a useful instrument in health care. Formally, an HTA is a broad assessment of the impact of a technology and is intended to include organizational, social, economic, and ethical considerations. Commonly, the focus is on the evaluation of a well-developed technology to identify the external effects and be able to choose between comparable technologies or alternatives for the existing situation. However, presumably mainly under the influence of policy pressure, HTAs generally are composed of clinical efficacy and cost-effectiveness analysis (CEA) studies, with every country adapting the HTA to its own needs (Reference Berg, van and Klazinga2;Reference Draborg, Gyrd-Hansen, Poulsen and Horder3;Reference Leys9). In addition to the characteristics of the technology, the purpose of the agency deciding on or financing the study is an important factor determining the exact HTA design. Battista (Reference Battista1) states that the complexity of the HTA has increased so much that input from other research fields is necessary to maintain its relevance.
In recent years, the need to fill in a gap in the approach of HTA became apparent. In 1995, Willems and Schade (Reference Willems and Schade22) confirmed this conclusion concerning issues that were related to general practice and pleaded for an approach that would take the dynamics of technology development into account. In addition to changes in the design of the technology, relevant domains such as practice organization and financing, patient reactions, and juridical and ethical aspects can change as well (Reference Berg, van and Klazinga2;Reference Oliver, Mossialos and Robinson11). To date, these domains were rarely covered in papers on HTA and especially the dynamics as such were not addressed (Reference Berg, van and Klazinga2–Reference Hummel4;Reference Lehoux, Tailler, Denis and Hivon8). Another reason for adaptation of the HTA follows from the fact that study periods can easily take 6 to 7 years from submitting the design to presentation of the results. A CEA, with its focus on effectiveness, seemed not to be sufficient to answer the scope of the questions related to the implementation of an evolving technology.
The issue of limited impact on the actual implementation and on coverage decisions was internationally confirmed and also related to the unrealistic paradigm of stability of both the technology and the environment, the ceteris paribus principle (Reference Oliver, Mossialos and Robinson11). Van Rossum (Reference Van19) demonstrated that, in several HTA studies initiated by the Dutch Health Care Insurance Board (CVZ), little visible effect could be found, especially related to the delay between the initiation of the HTA and the actual reporting of findings. As a consequence, practitioners started to develop their own guidelines for its use and even adapted the use of the technology guided by other studies that were published in the meantime, sometimes even immediately after the patient accrual of the original study ended. In 2005, the Centers for Medicare and Medicaid Services decided to provide the option for “coverage with evidence development” as a way out to make promising innovations accessible in an early stage. Instead of having to wait for the extensive, time-consuming process of generating evidence, early introduction is combined with obligatory participation in registration and research. This strategy asks for appropriate methods of technology assessment (Reference Tunis and Pearson16).
HTA studies that assume the technology and its environment to be stable are likely to produce outdated evaluations of the quality of the technology. The findings are outdated by the time the assessment is published; therefore, optimal guidelines cannot be developed to implement the technology. The focus of HTA studies needs to shift from studying the quality of a new technology to optimizing the technology's quality and effectiveness under dynamic circumstances.
More attention should be given to aspects of technology dynamics by acknowledging the sociodynamic processes and in that way influence the technology's development and implementation in a desired direction. A possible approach, constructive technology assessment (CTA), could ultimately lead to a more effective technology. CTA is based on theories of technology dynamics and attempts to influence technological design and implementation to improve the effectiveness of the technology in clinical practice.
CTA was first used in the 1980s outside the healthcare arena. Since its first use, CTA has developed from assessing the exact impact of a new technology to a broader approach, including the analysis of design, development, and implementation of that new technology (Reference Johri and Lehoux6;Reference Schot14;Reference Schot and Rip15).
The literature on technology assessment methods can be divided between diagnostic and intervention methods. Diagnostic methods of CTA include traditional social sciences techniques and also sociotechnical mapping techniques to identify the past and possible future scenarios of technological dynamics. Intervention methods are action techniques, including awareness initiatives, controlled experimentation, consensus conferences, and dialogue workshops, to influence technological development and application.
Only a limited number of papers have been published on methods of CTA as applied in health care. An example is the introduction of quality management as a management technology (Reference Van Harten17;Reference Van, Casparie and Fisscher18). Here, a combination of process analysis and outcome analysis was performed, using various methods that are common in social sciences. Another example is the use of systematic decision support as a tool to guide decisions that shape technology development and application (Reference Hummel4). An instrumental approach was successfully used to influence decision making to optimize the design of healthcare technology. A third example is a scenario approach that was used by Keesmaat (Reference Keesmaat7) to guide the development and introduction of a teleconsultation service for child physiotherapists based in a rehabilitation hospital. In the conceptual phase, an idea concerning the possible use of available information technology in teleconsultation for pediatric rehabilitation was projected into different scenarios for future services. This strategy was consequently used to guide the development in practice, and at present, a regional children's physiotherapy teleconsultation service is functioning successfully. In these studies, the concept of CTA proved to be feasible and was elaborated using different methods. In this study, we expand on the methodology of CTA based on these experiences. We will give a description of the research methodologies used for CTA in health care (intervention methods are outside the scope of this paper). A case study of CTA in clinical practice will be used to illustrate a possible CTA design and to discuss practical problems and possible solutions.
METHODOLOGY OF CTA IN HEALTH CARE
Phases of Technology Development and Relevant Research Aspects
To properly study the dynamics, CTA has to start before the new technology has been introduced into clinical practice. Normally, a technology is developed within a single, often scientific, organization or a network of organizations. Poulsen describes this as a development chain from initial invention, through test and retest procedures, to ultimate design and marketing in which, at all stages, an interaction with the (international) body of science exists (Reference Poulsen12). The various phases in clinical studies can very well be projected within this schedule. It can be argued that the innovators are singly or as a network acting as developers, whereas the actual implementation within the first number of other organizations—the early adaptors—starts thereafter, for instance with large phase 3 trials. Ideally, the CTA takes the dynamics of every separate phase into account.
The technology diffusion theory of Rogers can be used to relate the parameters for evaluation to the different phases that are characterized by the user groups of innovators, early adoption, early majority, late majority, and laggards (Reference Rogers13). A new technology that has recently been developed and is used by the first innovative users will need a different assessment approach than a technology that has been used by the early majority. For example, after clinical validation of its efficacy, the technology is assessed in only a limited number of patients during its first implementation and CEAs are commonly only performed when sufficient numbers of patients can be included.
CTA focuses not only on the technology but also on the environment in which the technology is introduced, and logically, the aspects that are to be studied in the different phases can vary along the assessment/implementation process. The continuous change of interaction between technology and environment during the different phases of diffusion can lead to changes in the aspects to be monitored per separate phase. Each transition of one phase into the other should, whenever possible, be marked with an evaluation. The first two phases of CTA will parallel the innovators and early adoption phase. When the early majority starts using the new technology, a prospective cost-effectiveness study can be performed. The end point of the CTA measurements has been reached when the quality of the new technology appears to be optimized and/or stabilization of its use has been achieved or (in the extreme) if the implementation is stopped because of serious concerns about the technology's quality.
Essential for the added value of CTA is a proven contribution to the (final) quality of the technology. In the literature on CTA and HTA, the concepts of quality and adequacy are both used, not just in terms of the Institute of Medicine definition (5), but also the ultimate impact of the technology in medical practice in a broader sense. In this paper, we will use the term quality. Depending on the technology at hand, the assessment has to focus on a mix of relevant aspects (see Table 1). These aspects are also known from the literature on HTA. All aspects should be taken into consideration for the assessment, but in the actual design, only those are included that are estimated relevant for the particular technology and environmental interaction. The nature of CTA makes it more likely that all relevant aspects will actually be covered.
Table 1. Aspects Studied in CTA
Note. Based on Poulsen (12) and the quality definition of the Institute of Medicine (5), Poulsen defined that a completeHTAshould at least include an integral assessment of clinical, economic, patient-related, and organizational parameters.Within these parameters different aspects can be distinguished.
Research Methods
The method and design of the exact research activities is determined by the nature of the technology (hard: a drug, a diagnostic procedure; soft: a management system), the stage of development and diffusion, and the aspects that are to be included in the study. In general, accepted methods of research in social sciences or health services research are used, but the combination of several additional or concurrent methods related to the various aspects is typical for CTA. For example, process analysis, patient satisfaction, and impact measurement and various forms of cost-effectiveness analysis can be used. Especially in the innovator and early adaptor phases, forms of action research can be appropriate. Often used approaches to study dynamic processes, however, are scenario methods.
Scenarios can be used to monitor the implementation of the technology through the various diffusion stages and to identify the need for intermediate evaluation or even interference through decision making or other “action research” -related techniques. Scenario methods, thus, can intentionally influence the process of the introduction of a new technology.
The relevant aspects that are identified and the diffusion phase in which it is to be executed will determine the method of research. For instance, in the early adopter phase, numbers of users of the technology are usually small and the technology use is unstable, so it is not possible to draw firm conclusions on cost-effectiveness yet. By giving feedback on the evaluated findings while they are collected, either by researchers, patients, or professionals, for instance on logistic aspects or effects on existing guidelines, more effective implementation can be enhanced.
Microarray Analysis: A Case Study
To illustrate the methodology of CTA, the controlled introduction of microarray analysis, as a new technology in clinical practice of breast cancer, will be described (see framework). With microarrays, we not only refer to the technique but also to the related logistic processes and procedures surrounding it.
The Dutch Health Care Insurance Board recently started a program to stimulate the controlled introduction of promising innovations in an early stage of development. By introducing the new technology in a carefully monitored program, microarray analysis would become available for an increasing number of breast cancer patients. CTA in this case study is aimed to ensure and improve the quality of the implementation of microarrays in clinical practice during the implementation of the technology in different health organizations.
In 2002, van't Veer et al. published the discovery of a gene expression profile consisting of 70 genes, using microarray analysis, that could predict survival chances in node-negative breast cancer patients better than current clinical and pathological factors at that time (Reference van't, Dai and van21). This 70-gene profile was first validated on breast tumor material of women who had breast cancer 10 to 20 years ago. Of these women, treatment information and follow up records were available. They found that the 70-gene profile had a high prognostic value (Reference Van, He and van't20). This technology enables us to assess the risk of distant recurrence within 5–10 years more accurately than using current clinical and pathological factors. Because of these promising results, the 70-gene profile has been developed into a prognostic test that can be implemented in clinical practice. To obtain a 70-gene profile after surgery, the tissue samples have to be preserved on dry ice or an RNA-later medium within 1 hour after the excision and sent to a central laboratory facility where the analysis is performed. Oncological surgeons or medical oncologists receive the 70-gene profile, indicating either a good or poor prognosis; they can use this result to shape decisions regarding the adjuvant systemic therapy policy (endocrine treatment and chemotherapy).
At the start of the study, the 70-gene profile was just developed and validated in retrospective series. This firm evidence from the validation was expected to be confirmed soon in a second (independent) retrospective validation. It was expected that it would take at least 10 years to bring this analysis into clinical practice when introducing it through the usual path of controlled prospective clinical trials. Therefore, it was decided that a controlled introduction would be appropriate, both from the viewpoint of the development phase of this promising innovation as well as from the position of the Dutch Health Care Insurance Board. In addition, it was considered important to establish whether a controlled introduction scheme would work. Because the technology was in an early stage (innovation/early adopter phase), the decision was made to start with a clinical pilot study.
The study will start with a maximum of six hospitals of varying size and nature to test the logistic impact, which can be described as innovators. In every stage of diffusion, the number of hospitals will be increased to reflect the natural diffusion process.
Operationalization of Aspects
According to the theory of CTA, this case study has to include all aspects relevant to measuring quality and all factors involved in the dynamics of design, development, and implementation of the technology. In this case study, those aspects are specified by (i) patient centeredness, (ii) user friendliness, (iii) timing, (iv) efficiency/efficacy, (v) juridical and ethical aspects, (vi) safety, and (vii) cost-effectiveness.
Based on the theory of sociodynamics, it becomes clear that all these aspects, in combination with the characteristics of the microarray analysis and the diagnostic process, can play a role in slowing down or stopping the implementation process. By studying these different aspects, clinical, economic, social, as well as organizational aspects are covered in this design. As CTA focuses on the sociodynamics of the new technology, these aspects are studied during the implementation process, special attention will be given to the changes that will develop in time. Along the different stages of CTA, the focus of the attention to the different aspects will most likely change.
Operationalization of CTA in the Phases of Development and Implementation
In the preparation phase of CTA (or zero base measurement) in this case study, information will be gathered on the nature of the microarray analysis and the organizational settings by means of literature research, documentation analysis, observations, and semistructured interviews with the professionals involved. This information will be used to propose changes in the development and implementation of the 70-gene expression profile. A process description will be made of the clinical practice of breast cancer care in the participating hospitals before and after introduction of the 70-gene expression profile. Based on these findings, a guideline for effective implementation of the prognostic tests using microarray analyses will be developed.
In the second phase, every participating hospital will be monitored for the actual application of the 70-gene expression profile. This finding will be compared with the preceding analysis and the guideline; important deviations will be observed, and feedback will be given. Once the implementation is realized in the concerned hospitals, data regarding the various aspects will be collected by repeated measures of documentation research, observations, tape recordings, and semistructured interviews with the professionals and patients involved. These results and data, in combination with the prospective validation of the microarray analysis, can lead to the ingredients for a cost-effectiveness study. A theoretical scenario for implementation and diffusion will be written. Based on recommendations from the earlier phases, some additional aspects could be studied, or existing aspects studied in more detail.
What points of evaluation will be used will be decided at the transition of the different phases of diffusion. When more knowledge is obtained about the different aspects of this new technology and its surroundings, more hospitals will be invited to participate. When numbers are large enough, a cost-effectiveness study will be started. The end point is the optimal implementation of the 70-gene profile using microarray analysis in clinical practice or can also be the decision to stop implementation.
DISCUSSION AND CONCLUSION
It is obvious that, so far, limited material is available on the various aspects related to the implementation of CTA in health care. It seems appropriate to apply CTA, especially under conditions of a promising technology in an early stage with uncertainty concerning its development course or an expected dynamic interaction between technology and environment. Especially for innovations that qualify for an early or conditional coverage decision, the use of CTA should be considered. However, there remain several issues that need to be addressed.
When exactly to choose CTA? CTA recognizes and uses the dynamics of technology implementation to optimize quality and clinical effectiveness. Through this dynamic view, CTA researchers are able to react to changes that are made, intentionally or unintentionally, to the technology, or to the environment surrounding the technology. CTA cannot claim to be the single method to cope with dynamic circumstances. Bayesian methods are also meant to deal with this aspect; however, a basis of data considering (cost-)effectiveness is required for this purpose. This requirement makes application in the innovation and early adaptor phases less likely. Recently, however, some publications reported experiences with probability statistics in predicting organizational change (Reference Molfenter, Gustafson, Kilo, Bhattacharya and Olsson10). This ability requires a sufficient body of knowledge concerning predictive factors, and these data will commonly not be available in early stages of diffusion.
Clinicians will obviously advocate methods that are primarily focused on effectiveness; classic designs yield the most evidence and have more of a chance to be published. Challenging the fact that it might not be self-evident that there is a ceteris paribus situation in a discussion on the study design, thus, leads to presumptions about the possible use of the results. Obtaining commitment from clinicians is thus a major issue as they will favor classic HTA approaches. In the case study, CTA starts at an early stage of the implementation process and the uncertainties about the exact implementation course made it easier to involve the clinical team.
It needs to be further studied for which technologies CTA is most suitable. It seems obvious that drug-related research is more likely to follow classic research designs, whereas technologies involving various technology domains or those interacting intensely with the environment might be candidates for CTA. Experiences of the researchers involve hard technologies, such as an assistive heart pump and a voice prosthesis; mixed technologies, such as e-consulting; and a soft technology, such as quality management. Our case study reveals a few relevant considerations in designing a CTA methodology. These points of consideration relate to the contents, objectivity, and timing of the CTA analysis.
Phases and Aspects
We defined which aspects can be studied in a CTA analysis and presented these in a case study. The exact timing of studying those aspects relates to the different implementation phases, as described by Rogers (Reference Rogers13). The choice for the aspects to include in the CTA can be based on evidence found in the literature and the scenario analyses. For instance, in our case study, we have decided that juridical and cost-effectiveness studies will be conducted to closer study further along the implementation process, while patient acceptability and timing will be two of the first aspects to be investigated. At every point of evaluation, the aspects studied have to be carefully looked at and a decision has to be made on which aspects to study in the next phase of the implementation process. As such, the choice of aspects will not be a decisive difference; the paradigm of technology dynamics makes it more likely that a comprehensive approach will be upheld during the CTA process.
Objectivity in View of Participative Observation and/or Action Research Approaches
Studying technologies that can influence daily routines requires methods that go into sufficient depth to actually verify the nature of changes in sufficient detail. Although guidelines exist for clinical processes, the actual clinical practice may deviate from these guidelines. These sometimes seemingly innocent deviations may be very important for a smooth introduction of the new technology. This finding makes it important to involve clinical practitioners in CTA but also to guarantee sufficient insight of researchers into daily practice.
To optimize flexibility and awareness of the researcher for changes in the circumstances, close interaction with the clinical team may thus be necessary. This contact can influence clinical practice, for instance, by asking feedback on results already during the implementation process. Obviously, this process can also influence the interpretation of the findings by the researcher. Objectivity is an issue and needs to be guaranteed, for example, by the use of standardized measurements and verification by other researchers in each stage of the CTA. Especially in the absence of traditional designs, mostly during the earlier introduction phases, the risk of subjectivity has to be compensated.
Phases and Continuation
It is most logical to consider a change of methodology or focus of the technology assessment when a new diffusion phase is starting. Especially the prospect of stabilization of circumstances or use and the possibility to involve larger numbers of patients enabling studies that reach enough power within reasonable time should lead to evaluation. The involvement of CTA in the implementation process stops when the quality of the new technology appears to be optimized and/or stabilization of its use has been achieved or (in extreme) if the implementation is stopped because of serious concerns about the technology's quality. In practice, this end point will not always be clearly marked.
With attention to the issues addressed, CTA seems an appropriate method to evaluate the introduction of a new technology, because it combines speed with carefulness. This strategy can give the impetus to a more direct influence on policy making, especially as agencies increasingly tend to experiment with early coverage decisions. In addition to the HTA as we know it, CTA can be a complementary approach, especially to guide the introduction of technologies in a controlled way.
CONTACT INFORMATION
Kirsten F. L. Douma, MSc (k.douma@nki.nl), PhD student, Kim Karsenberg, MSc (k.karsenberg@nki.nl), Research Assistant, Division of Psychosocial Research and Epidemiology, The Netherlands Cancer Institute—Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX, The Netherlands
Marjan J. M. Hummel, PhD (J.M.Hummel@bbt.utwente.nl) Assistant Professor, Faculty of Management and Governance, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands
Jolien M. Bueno-de-Mesquita, MD (j.bueno@nki.nl), PhD Student, Department of Pathology, The Netherlands Cancer Institute—Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX, The Netherlands
Wim H. van Harten, MD, CommMed, PhD (w.v.harten@ nki.nl), Chair, Quality Management of Health Care Technology, Faculty of Business, Public Administration and Technology, University of Twente, P.O. Box 217, 7500 AE Enschede, The Netherlands; Member Executive Board of Directors, Director Organization & Management, Department of Psychosocial research and Epidemiology. The Netherlands Cancer Institute—Antoni van Leeuwenhoek Hospital, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
