The establishment of health technology assessment (HTA) agencies by national or regional supervisory authorities, insurance companies, hospital networks, or medical associations has become widespread, especially since the 1990s with the creation of the first international network of HTA agencies in 1993: the International Network of Agencies for Health Technology Assessment (INAHTA) (13). This network develops shared methods for producing HTA reports designed to provide objective information on the medical, economic, and organizational value of new technologies (evidence-based health policy) for healthcare professionals and decision makers.
Assistance Publique–Hôpitaux de Paris (AP-HP) is a network of 39 French university hospitals located in the Paris region. Since 1982, AP-HP has set up an HTA agency called CEDIT (Comité d'Evaluation et de Diffusion des Innovations Technologiques—Committee for the Assessment and Dissemination of Technological Innovations), which is part of the AP-HP's Medical Policy Directorate. CEDIT is one of the founding members of INAHTA. Its special area of skill relates essentially to innovative medical devices and procedures. The various players involved in health decision-making processes within the hospital network are entitled to seek an opinion from this agency: users (healthcare providers) or people involved in making policy or financial decisions (senior management of hospitals and Central Office, purchase division). Once an evaluation has been carried out according to the HTA general methodology, CEDIT issues recommendations on the value of the technological innovation and practical advices on its dissemination within AP-HP hospitals. Furthermore, it may consider that supplementary investigation is necessary and provide methodological, logistic, and financial support for the purposes of carrying out such investigations. Decision makers retain full freedom as to whether to follow, discuss, or simply take note of the recommendations.
Producing valid scientific information is necessary, but it is not enough to promote any given technology within the health system (4). Hence, it is necessary to assess how the work conducted by HTA agencies is put to use and what its impact is on professional practice, health provider prescriptions, and on the introduction of new technologies into clinical practice. Several studies have shown that recommendations have a major impact (12;15), but as far as we know, none has been carried out in France concerning the assessment of technological innovations. It is difficult to extrapolate the results of surveys carried out in other countries, because health systems vary considerably from one country to another. Furthermore, it is generally difficult to accurately assess impact because of the variety of players involved, the range of potential types of impact, and the many influencing factors (15). Because of its status as part of the hospital network, CEDIT may be an interesting model for surveying the impact of recommendations (8). Indeed, it is easy to compare technologies actually acquired by the hospitals compared with the relevant recommendations on dissemination. The goal of our investigation is to examine the extent to which CEDIT's recommendations match user's expectations and to assess their influence on decision-making.
METHODS
Two studies—one using semidirective interviews and the other using case studies—were conducted between March and April 2002 by a public health resident who spent 6 months in training at CEDIT.
Interviews were conducted among the various players affected by recommendations on the dissemination of a technological innovation. We chose fourteen interviewees to include both those who had and those who did not have implications with CEDIT's work. Five of them worked at AP-HP Headquarters: the Secretary General, a finance division officer, a purchase division officer (General Agency for Health Equipment and Products, AGEPS), a policy officer from the medical policy directorate in charge of supervising Hospital Medical Plans, and the SAMT (biomedical and imaging activity department) manager; the other nine respondents worked in one of the thirty-nine hospitals: three establishment managers, two biomedical engineers in charge of equipment plans, three hospital practitioners, and a chairman of a Medical Advisory Board. The doctors (anesthetist, cardiologist, and radiologist) and the biomedical engineers were chosen on the basis of their specialty and their hospital. The topics dealt with during the interviews were mainly the system for disseminating the recommendations, knowledge of how CEDIT operates, and the impact of the recommendations. Respondents' opinions are commented on in the results section.
A second study was conducted on the basis of case studies focusing on the dissemination of the technologies in the wake of those recommendations. The recommendations published between 1995 and 1998 were chosen so as to have a long enough time lag between the publication of a recommendation and its implementation. Recommendations arising from supplementary surveys to be performed during that period were not included. The assessment of the dissemination status of a technology and the influence of the recommendation on the introduction of that technology was performed by a survey in the hospitals (equipment inventory, purchase division contracts, opinions of healthcare providers). Where applicable, consideration also is given to the recommendation update subsequent to the study period that may have had an impact on the dissemination status of the technology.
RESULTS
Image and Awareness of CEDIT
CEDIT has a good scientific image particularly in the eyes of administrative staff. Some practitioners are sometimes disappointed by the topics dealt with, which they consider not sufficiently innovative.
Awareness by the various interviewees of how CEDIT operates is variable and often incomplete, but they believe they are informed about what is relevant to them. The main area of criticism is the time lag occurring between a request for assessment and the publication of the recommendation. CEDIT's particular position, that is, close to the decision-making departments of AP-HP, is sometimes criticized.
Perception of Recommendation Dissemination
The dissemination procedure for recommendations is generally considered to be good. CEDIT disseminates its recommendations to decision makers and users (general secretary of AP-HP, finance division, purchase division, medical policy division, establishment managers, biomedical engineers, chairmen of medical advisory boards, and heads of medical, surgical, and pharmacy departments). The recommendations are sometimes disseminated second hand locally in hospitals to particular targets (senior nursing staff, department heads, and so on). This secondary dissemination route sometimes lacks efficiency and, therefore, generates a negative image of the CEDIT and an underutilization of its recommendations. There is little awareness of CEDIT's Web site was launched at the end of 2001, as opposed to the boxed text presenting CEDIT's recommendations that is published in the monthly AP-HP newsletter sent to doctors.
Interest in the Recommendations
The recommendations are most often perceived as being clear and well drafted and generally arouse interest. The respondents essentially read recommendations that relate to their area of activity. Some people, often members of the administrative staff, read only the paragraph that summarizes CEDIT's opinion on disseminating the technology. The medical policy directorate's officers, the biomedical engineers, and some heads of medical department (in particular the Medical Advisory Board chairmen) read all the recommendations and sometimes the report also. Only one person considered conclusions to be insufficiently assertive. Conversely, one hospital manager believed that this type of formulation allows some welcome freedom for hospitals.
Grounds for Requesting an Assessment
The grounds that encourage the various players interviewed to request a technology assessment by CEDIT vary. Members of the administrative staff tend to seek an opinion from CEDIT so as to get an objective presentation of the technology, data on the financial impact, and a relevant opinion as to whether it should be disseminated. They appreciate the scientific quality of these opinions. They believe that they are often the instigators of a request for assessment, sometimes indirectly, by encouraging healthcare providers to seek an opinion from CEDIT after discovering a costly activity within the hospital.
The doctors interviewed state that they tend to seek an assessment from CEDIT essentially for very expensive innovations for which hospital funding is difficult to obtain in the hopes of obtaining a tool for negotiating with their management, especially if several departments are competing for budget appropriations. These recommendations enhance their credibility when requesting the acquisition of new technologies. Some healthcare providers say that they request an assessment from CEDIT with the aim of being allocated logistic, methodological, and financial resources for the implementation of clinical studies. In this case, they believe it is then easier to continue to use the technology at the end of the study because it is already installed and staff have been able to acquire some experience. When manufacturers or special research allocations allow easy access to a technology, they prefer not to seek the opinion of CEDIT to avoid the risk of a negative recommendation. Finally, the study shows that the influence of CEDIT's recommendations on some specialties such as intensive care is very little.
Implementation of Recommendations
All of the respondents would like to see compliance with the recommendations, particularly when they are negative. In the case of a positive recommendation, it is examined with respect to hospital priorities. The dissemination of major innovations that improve the quality of care appears to be unconnected with the recommendations. Respondents consider that there is a risk of noncompliance with CEDIT's recommendations that are published once the technology has already been implemented. Only one respondent expressed the wish that recommendations should be made compulsory.
The AP-HP financial division states that recommendations are taken into account in certain particular areas that involve a considerable financial component for all hospitals taken together for the purpose of granting special funding. The recommendations are followed as far as possible by the purchase division to establish the markets. They are also sometimes used in applications for authorization from national supervisory bodies—in particular for applications from the departments that deal with “major” medical equipment (SAMT).
The hospital Medical Advisory Board sometimes uses the recommendations to set out an equipment plan and define the objectives and resources commitments. When a recommendation is anticipated, purchases are often deferred, particularly if the technology is expensive and/or there are still doubts about its value. But in some cases, technology has been disseminated despite a recommendation to the contrary. Hence, when a study is ongoing, the technology is sometimes disseminated before the publication of the final recommendation.
The detailed results of the case study are provided in Table 1. Of the thirteen recommendations analyzed, five were the outcome of a supplementary study requested by CEDIT. Seven recommendations certainly had a major or considerable impact, in particular as a result of subsequent financial decisions (funding provided or withheld for the purchase of the technology). One of these recommendations (mechanical ventricular assist devices), however, was not entirely complied with, because the technology had been disseminated in one additional center compared with the recommendation. For three recommendations, it is difficult to distinguish between the impact of the recommendation and that of the experience acquired by the teams during the period the supplementary study was conducted according to CEDIT recommendations. The impact of these three recommendations is deemed to be moderate. The dissemination of one technology (video enteroscope) does not comply with the relevant recommendation. Finally, two recommendations on the introduction of technologies were impossible to appraise because of the influence of major external factors.
DISCUSSION
The studies based on semidirective interviews and case studies presented here have shown that CEDIT's work has a fairly considerable impact and that it is useful to decision makers at AP-HP. Furthermore, the study based on interviews shows that the respondents' opinions and use of CEDIT's work varies depending on the respondent's function: decision-making aids for administrative staff, negotiating tools for users of the technologies. One of the limitations of our research derives from the small sample of respondents that is not fully representative. Nonetheless, all categories of staff that are involved in decision-making processes in hospitals and, therefore, concerned by CEDIT recommendations were interviewed.
Clearly, the introduction of new technologies is not always based on an analysis conducted by an HTA agency (20). The grounds for establishing the need for an overall assessment are closely connected with the impact that the new technology may have in medical (new diagnostic tool or treatment), economic (high unit cost or volume effects deriving from the size of the target population), organizational (impact on care management), ethical and/or legal terms, or even on policy (public health priorities) (7). From our study, the main criterion appeared to be cost; hence, some medical specialties are more concerned by CEDIT's work than others—that is, cardiology and medical imaging as opposed to intensive care, an area where recent innovations are not costly.
The major criticism directed against CEDIT is the time taken to complete investigative procedures. These time requirements derive from the very principle of HTA that relies on a very stringent but time-consuming methodology. In comparison with the rapid development of technologies, especially those relating to medical devices, long-winded HTA procedures can be seen as a hindrance to the introduction of new technologies. However, it should be taken under consideration with regard to the time needed for research and development of technology by manufacturers. Furthermore, lengthy HTA processes can be mitigated by establishing fast-track procedures for specific, well-defined questions (12;17). Indeed, CEDIT established a rapid HTA procedure at the beginning of 2004.
Our study has shown a certain lack of visibility associated with CEDIT's independence with respect to regulation procedures. Some respondents considered CEDIT to be too closely connected with decision-making departments. This finding is a reflection of the ambiguity of HTA agencies whose credibility and objectivity must be guaranteed by their independence with respect to suppliers of course but also with respect to the users of medical technologies and their funding bodies. Its role is to supply the material for appraisal but not to take part in decisions (6). Decisions subsequently need to take into account, at the local level, previous investment, financial constraints, and the current level of activity and sometimes the medical strategy implemented in the hospital (5). However, recommendations on the introduction of new technology increasingly include recommendations on clinical practice, particularly in the case of highly specialized technologies, for example, when it is desirable to set up a special monitoring procedure for patients (such as a registry) or to designate specialized consulting centers. Implementation of such recommendations is likely to come up against the same obstacles as guidelines on clinical practice. It has indeed been shown that simply publishing the latter is often not enough to bring about changes (9–11;16), and several factors and procedures that encourage implementation of clinical guidelines have been identified (21). Furthermore, HTA recommendations that include considerations relating to practice will increasingly require systems of regulation both to facilitate the introduction of the technology in the ad hoc conditions and to ensure that they are abided by. In this instance, it may not be advisable to separate the structure that identifies new technologies from the one that is in charge of disseminating knowledge and overseeing implementation.
The assessment of the impact of CEDIT's recommendations on decision makers choices and on the introduction of new technologies conducted by means of a study based on interviews and case studies shows that, overall, there is a good match between the recommendations issued and their implementation. The main factor that fosters compliance with the recommendation appears to be the existence of a system of regulation, as in the case of guidelines on clinical practices (21). The regulation method may be of a financial nature when the technology is particularly expensive: additional budgetary restrictions (1) or absence of funding when a negative recommendation is issued generally act as a hindrance to the introduction of a new technology. Conversely, allocation of special appropriations acts as a strong incentive to the introduction and dissemination of technologies (e.g., cochlear implants and implantable phrenic stimulation in our case studies). Regulation by rules and regulations also encourages application of the recommendations. In France, this is true in the case of “major” medical equipment (especially medical imaging equipment), which requires a ministerial approval granted in accordance with a “health map” defined on the basis of regional needs (currently undergoing revision). Other factors could influence the extent to which a recommendation is followed. For example, the existence of major external constraints limits the dissemination of the new technology—this is the case when a particular technical environment is required that is not available in establishments (e.g., the case of the Thoravision system, which at best is used in conjunction with picture archiving and communication systems [PACS]). Moreover, undertaking a supplementary investigation often supports the respect of a recommendation, but in this case, it is difficult to distinguish between the impact of the experience acquired by the teams that conducted the latter study from that of the recommendation deriving therefrom.
Our study highlights three main reasons for failure to follow recommendations. First, with time, the recommendation may become obsolete as a result of developments in knowledge and in the technology. HTA results can only be considered applicable for a given technological development at a given stage of progress in medical engineering and in a given context. That is why it is essential for HTAs to be repeated. The problem, however, is to establish exactly when the technology, context, and/or scientific knowledge have developed sufficiently to warrant a further evaluation, as pointed out by Mowat et al. (18). The case study that provides historical data on the CEDIT recommendations investigated shows that several recommendations were reassessed, and consequently, dissemination of the technology was in tune with the latest recommendation (e.g., cochlear implants). In contrast, several hospitals acquired an electronic video enteroscope despite the recommendation advising that use be restricted to the existing stock at the time. An increased level of activity might explain that more hospitals subsequently acquired the technology, but this explanation has not been verified.
The second reason for the mismatch between the recommendations and practice suggested by the interviews arises from poor knowledge of the recommendations. The dissemination procedure established by CEDIT seems relatively effective. However, the information is disseminated when it is produced, whereas application may be deferred. It is essential, therefore, for earlier work to be readily accessible. In this respect, the creation of the CEDIT's Web site is a major asset.
Finally, the third reason for the mismatch between a recommendation and practice derives from the time lag required to complete a supplementary study with regard to a technology that is developing rapidly. As a result, a new technology is sometimes disseminated before proof of its value—in terms of (cost) effectiveness or even efficiency—is provided, as described in the case of cardiac pacemakers (2).
Our work has highlighted the difficulties that exist in some cases in establishing the impact of CEDIT's work on the introduction of new technologies. This difficulty is particularly true in the case of technologies whose value becomes apparent after several years of use. In such cases, it is difficult to say whether the dissemination of the technology would have been the same in the absence of an HTA recommendation (e.g., automatic contrast medium injector for magnetic resonance imaging). Assessing the impact of some recommendations can be made difficult by the existence of a particular set of circumstances that fosters the introduction and dissemination of the technology (e.g., the case of outpatient diagnosis of sleep apnea due to shortage of staff to carry out a diagnosis in conventional hospital conditions). Other difficulties could arise such as industry developments that lead to technological options been withdrawn or, conversely, imposed without it being clear whether or not the HTA recommendations in any way affected these developments. Furthermore, it is often difficult to distinguish between the impact of the recommendations as such from other sources of information (congresses, scientific publications, and so on), just as in the case of recommendations on professional practices (9).
Our results can be compared with impact studies conducted by other HTA agencies. Hence, the CETS—the Health Technology Assessment Council in Quebec—estimated in 1993 that, of the seventeen recommendations studied, six had a major influence, seven a moderate influence, one a slight influence, and three had no influence at all (14). In 1997, considerable impact was found for twelve of the sixteen recommendations studied (15). Hailey et al. (12) measured the impact of rapid HTA in response to an urgent request. This study conducted among the players concerned by the recommendations also highlighted considerable impact and influence on policy decisions. In 2003, the British National Health Service reported that their work had considerable impact on regional policy decisions and on local clinical practices on the basis of a before/after type study (3). Very recently, the National Institute for Clinical Excellence, on the other hand, reported that its work had a very moderate influence on general practitioner prescribing (23). Similarly, Vermeulen et al. (22) stated that HTA has little influence on decisions relating to health policy in Belgium, which they attribute to the fragmentation of decision-making and application centers and the strong influence of lobbies (patients associations and the pharmaceutical industry). Furthermore, Oliver et al. (19) stress that some players are skeptical with respect to the value of work done by government agencies, which are perceived as being essentially political, and fail to properly consider practical aspects. All these experiences together with our work suggest that the impact of HTA on practices and the introduction of new technologies is higher the more circumscribed is the target of the recommendation.
CONCLUSION
On the basis of a study using semidirective interviews and case studies, the HTA recommendations established by a body attached to a hospital network, CEDIT, were found to have considerable impact. Despite the rather small sample of persons interviewed, the original feature of this study lies in its qualitative section, which highlighted that there is a good match between CEDIT's objectives and its assignments. An assessment of the impact of these recommendations on the introduction of technologies uncovered the main factors that foster compliance with recommendations in the area of clinical practice that could be material for future investigations.
CONTACT INFORMATION
Florence Bodeau-Livinec, MD (livinec@vjf.inserm.fr), Public Health, at the CEDIT's, Scientific Secretariat, Department of Medical Activities, Assistance Publique-Hôpitaux de Paris, 3 avenue Victoria, 75004 Paris, France; Researcher in Epidemiology, U149 Research Unit on Perinatal Health and Women's Health, INSERM, 16 avenue Paul Vaillant-Couturier, 94807 Villejuif cédex, France
Emmanuelle Simon, PharmD, PhD (emmanuelle.simon@ sls.aphp.fr), Manager of the CEDIT's Scientific Secretariat, Department of Medical Activities, Assistance Publique-Hôpitaux de Paris, 3 avenue Victoria, 75004 Paris, France; Office of Licensing and Technology Transfer, Assistance Publique-Hôpitaux de Paris, Hôpital Saint-Louis, 1 avenue Claude Vellefaux, 75475 Paris cédex 10, France
Catherine Montagnier-Petrissans, PharmD (catherine.montagnier-petrissans@sap.aphp.fr), Manager of the CEDIT's Scientific Secretariat, Department of Medical Activities, Assistance Publique-Hôpitaux de Paris, 3 avenue Victoria, 75004 Paris, France
Marie-Eve Joël, PhD (joel@dauphine.fr), Professor in Health Economics, Unit of Research in Health Economics, Dauphine University, Paris IX, Place du Maréchal de Lattre de Tassigny, 75775 Paris cédex 16, France
Elisabeth Féry-Lemonnier, MD (elisabeth.fery-lemonnier@sap.aphp.fr), Head of the Department, Department of Medical Activities, Assistance Publique-Hôpitaux de Paris, 3 avenue Victoria, 75004 Paris, France
The authors acknowledge the comments on the study given by Nathalie Jakobi-Rodrigues, Jean-Philippe Perrin, and Emmanuel Charpentier of the Comité d'Evaluation et de Diffusion des Innovations Technologiques, Assistance Publique-Hopitaux de Paris, France.
