Health technology assessment (HTA) has been established as a useful tool to inform decision makers about the real value of a specific health technology (HT) in a given healthcare system (Reference Lafortune, Farand, Mondou, Sicotte and Battista1). Some states have set-up a national institute for HTA, such as the National Institute for Healthcare Excellence (NICE) in England, the Haute Autorité de Santé (HAS) in France, or the Swedish Council on Technology Assessment in Healthcare (SBU) in Sweden. These institutes produce reports summarizing an extensive assessment of a given HT, which takes months if not years to complete.
University hospitals are usually the entry point for new HTs. As they are frequently more expensive than existing treatment, their introduction may lead to financial imbalance if it is not adequately managed. To assist the hospital directorate in making a sound science-based decision about introducing a new HT, some hospitals created hospital-based HTA (HB-HTA) units. These units allow producing timely HB-HTA reports, tailored to the hospital context (Reference Sampietro-Colom, Lach and Cicchetti2). However, the impact of such HB-HTA reports has seldom been studied. A recent systematic review (Reference Gagnon, Desmartis, Poder and Witteman3) including eighteen studies showed that only four of them reported the impact of the HB-HTA reports and recommendations on hospital decision making. Two of these studies were carried out in Canada: one over 5 years at a surgical department in Calgary (Reference Poulin, Austen, Kortbeek and Lafrenière4), and the other one over 8 years at a teaching hospital in Montreal (Reference McGregor and Brophy5). The other two studies were carried out in Europe: one from the HB-HTA unit (CEDIT) covering nineteen university hospital in the region of Paris, France (Reference Bodeau-Livinec, Simon, Montagnier-Petrissans, Joël and Féry-Lemonnier6), and the other one from the Ludwig Boltzmann Institute (LBI-HTA) serving the whole hospital system in Austria (Reference Zechmeister and Schumacher7).
In Switzerland, no national institute for HTA exists, but Lausanne University Hospital created a HB-HTA unit in 2002. After 10 years of activity, we wanted to assess the HB-HTA process's utility as perceived by our clinicians, and the accuracy of the HB-HTA reports and recommendations over time.
METHODS
All HB-HTA reports published between 2002 and 2011 by our HB-HTA unit were retrieved. For each HT accepted by the hospital management, a semi-structured interview with the clinician responsible for using the HT was conducted by an investigator not involved in the HB-HTA process (XG). The structure of the interview is displayed in Table 1. The interview aimed at assessing (i) the utility of the HB-HTA process as perceived by the clinicians; (ii) the accuracy of the expected medical impact of the new HT as described in the HB-HTA report; (iii) the compliance with the indications for treatment as defined in the HB-HTA report at the time of assessment. The interviews were recorded and reported by the investigator on a separate file for each HT, checked for completeness and accuracy by the senior author (JBW), and summarized in a tabular format. These results were used by the investigator to fulfil the requirements of his master degree in medicine (internal report available in French).
Table 1. Semi-structured Interview Guide: Retrospective Assessment of the HTA Process
The first objective, the utility of the HB-HTA process as perceived by the clinicians, was assessed qualitatively on the basis of the clinicians’ answers to the questionnaire.
The second objective, the accuracy of the expected medical impact of the new HT as described in the HB-HTA report, was assessed quantitatively and qualitatively. The administrative data of patients treated in the year 2012 with each HT included in the study were retrieved from the hospital information system. Number of patients, type of setting (inpatient or outpatient), length of stay (LOS), price of the HT described in the HB-HTA report at the time of assessment were compared with the 2012 patient data. They were rated as concordant for ratios ±20 percent, and otherwise higher or lower. The accuracy of the expected success and complication rates were assessed qualitatively during the interviews with the clinicians.
The third objective, the compliance with the indications for treatment as defined by the HB-HTA report at the time of assessment, was assessed qualitatively. The evolution in the indications for the new HT was assessed during the interview with the clinicians, and categorized into “same,” “broadened, “restricted,” or “modified.” We finally examined whether there was a correlation between the concordance in patient numbers (concordant, lower of higher) and the evolution in indications.
RESULTS
Over the 10-year period, forty HB-HTA reports were issued, of which thirty-four (85 percent) were approved by the hospital management for clinical use. The list of assessed HTs is displayed in Table 2, together with the year of assessment and the decision of the hospital management.
Table 2. List of HTA Reports with Their Most Important Characteristics
Utility of the HB-HTA Process as Perceived by Clinicians
Out of the twenty-eight clinicians responsible for using these thirty-four HTs, twenty-seven accepted the interview; 85 percent of them (23/27) acknowledged the utility and necessity of the HB-HTA process. For thirteen of them (48 percent), continuous improvement in care provided to patients justified that the hospital takes action to introduce new HTs. For six of them (22 percent), the HB-HTA process provided a welcome frame and support in case of serious complications experienced with the use of the HT. For six of them (22 percent), the HB-HTA process mandated creating protocols of use and assessing the clinical impact of the HT, which were favorably considered. Budgetary impact assessment was mentioned as a limiting factor only by four clinicians (15 percent). A negative opinion was expressed by only four clinicians (15 percent), who considered the process as useless and an additional hurdle set-up by the hospital administration. However, the HB-HTA reports remained rather confidential, as they were not diffused to their colleagues inside the involved clinical service by twenty-two of the twenty-seven responsible physicians (81 percent).
Accuracy of the Expected Medical Impact of the New HT as Described in the HB-HTA Report
Of the thirty-four HTs adopted, five (18 percent) were not in use in 2012 (Table 2): two had been withdrawn (ExPRESS tube for glaucoma, photodynamic therapy for bronchial and esophageal cancer), two replaced by another HT of the same nature (sirolimus stent, left ventricular assist device), and one was awaiting introduction (suture-less aortic valve). Of the twenty-nine remaining HTs, the observed number of patients treated in 2012 was in accordance with expectations (±20 percent) in eight cases (28 percent), higher than expected in four cases (14 percent), lower in fifteen cases (52 percent), and not available in two cases (7 percent). Detailed distribution of the observed versus expected numbers of patients are displayed in Table 2.
A modification in the setting of treatment was observed in four cases: in two of them, when both outpatient and inpatient treatments were planned, but inpatient treatment was preferred (endocavitary guidance for foramen ovale closure, cryotherapy for cancer). On the other hand, a shift to outpatient treatment was observed in two other cases, for which it was considered as possible (Spyglass cholangioscopy, liver radioembolization).
LOS could be assessed in eighteen of twenty-nine HTs (67 percent). For eleven of them, the LOS was not reported in the HB-HTA report, as the new HT was expected to have no impact on it. For the seven others, the observed LOS in 2012 was higher than forecasted by an average of 61.2 percent.
Price of HTs was available in fourteen of twenty-nine HTs (48 percent). The price of devices observed in 2012 was higher than expected for nine of them (64 percent) and lower for five of them (36 percent). Most of these variations were linked with fluctuations in devices’ prices.
Two HTs had a higher complication rate than expected. For one (Biopatch with chlorhexidine), complications were mild and did not prevent its use, while for the other (Sunchip system for hypothermic intra-abdominal chemotherapy), complications were severe and limited its use to selected cases.
In three of twenty-nine HTs (10 percent), a diminution in expected success rate was reported. In two cases (Green light PVP laser and Superdimension bronchoscopy system), pre-existent alternatives were preferred, while for the third one (cerebral micro-diagnosis for patients with traumatic brain injury), its use was not questioned despite an observed failure rate of 20 to 30 percent, as no real alternative exists.
Finally, we discovered that two HTs were not used as forecasted because the necessary staff increase had not been accepted by the hospital directorate (Propofol use in endoscopy and hypnosis in major burns).
Compliance with the Indications for Treatment as Defined in the HB-HTA Report at the Time of Assessment
Indications as mentioned in the HB-HTA report were initially followed in 26 out of twenty-nine HTs (90 percent). For the past three, technical difficulties limited its use in one HT (Microporus stent grafts for treating cerebral aneurysm), while the infrastructure necessary for using the Superdimension lung navigation system for bronchoscopy was not available for all patients as initially planned. Finally, compliance with indications could not be assessed in the third HT (sutureless aortic valve replacement), as it had not been introduced at the time of the study.
Over time, indications changed in sixteen of twenty-nine HTs (52 percent). Indications were broadened for six, and restricted for seven of them. For the last three of them, a modification of the spectrum of use by the addition of another HT or a change in treatment modality was observed. Detailed results for each HT is displayed in Table 2. There was no correlation between the evolution in indications overtime and the concordance in forecasted versus observed patient numbers.
Finally, clinician preferences seemed to play an important role. Clinicians taking over responsibility for the field in which their predecessors had required the introduction of a given HT did not always share the same opinion about the utility of this HT (one example was the Superdimension bronchoscopy system). As a consequence, the HT was not used as initially forecasted.
DISCUSSION
This study allowed assessing the impact of a HB-HTA process and unit at a university hospital, and found that the accuracy of HB-HTA reports forecasts was limited, especially in the long-term.
Although such studies are still scarce, four of them were previously published, but each with a somewhat different focus. Poulin et al. (Reference Poulin, Austen, Kortbeek and Lafrenière4) retrospectively assessed the 5-year HB-HTA activity of a unit located in a surgical department in the Calgary Health Region in Canada, and described the difficulties linked with the approval or rebuttal of 68 clinician requests, so that conditional approval appeared as the best way to avoid selecting and investing in inefficient HTs.
In Canada as well, but in Montreal, McGregor and Brophy (Reference McGregor and Brophy5) summarized the 8-year experience of the HB-HTA unit at McGill University Centre, focusing on the budget impact of the decision process in fifty-five requests. Rejecting nearly half of them allowed saving amounts up to several million dollars per year.
In France, Bodeau-Livinec et al. (Reference Bodeau-Livinec, Simon, Montagnier-Petrissans, Joël and Féry-Lemonnier6) assessed the activity of the CEDIT (French translation of the Committee for the Assessment and Dissemination of Technological Innovation), serving the thirty-nine university hospitals in the Paris region. Of thirteen HB-HTA reports produced over a 4-year period (1995–98), ten had a definite impact on the adoption or rebuttal of the HTs by the hospitals’ directorates.
Finally, in Austria, the Ludwig Boltzmann Institute for HTA, responsible for HB-HTA for the hospitals of the whole country, published forty-two rapid HTA reports over a 10-year period, with a high rate of negative recommendation (72 percent). Even if hospitals only partly followed the recommendations (in approximately 50 percent of the cases), they saved again several million € over the whole period.
As compared with these other four studies, our adoption rate of 85 percent was rather high. Therefore, our study design focused on the practical impact of adopted HTs in an effort to identify the unwanted consequences of this hospital policy. We discovered several of them, which will be discussed below.
First, our study showed that follow-up of the impact of HB-HTA reports was hampered by technical difficulties in retrieving data for thoroughly assessing the long-term outcome of these new clinical practices. For example, available data allowed assessing LOS in only eighteen of thirty-four HTs (53 percent). A specific effort to collect appropriate data is unavoidable if close monitoring is contemplated.
The matter is further complicated by the fact that initial projections can be subjected to several distortions. Most estimates tended to be over-optimistic about the number of patients likely to be treated by the new HT and the corresponding LOS. Then, the success rate of a new HT was sometimes limited by its impact on the availability of hospital's infrastructure and/or human resources. These two aspects should be assessed at the same time as the HT to grant a global approval before HT introduction.
Furthermore, clinical practice is relentlessly evolving, so that indications and physicians’ preferences tend to change over time, explaining some of the discordances found between forecasted and observed patient numbers. This reality should be incorporated into the impact assessment of any new HT on a long-term basis. Revision of indications and recommendations at regular intervals should be carried out to limit this kind of surprise.
On the other hand, the finding that the acknowledgment of the HB-HTA process's utility among clinicians was quite high (85 percent) is encouraging. This figure is very similar with the findings of two earlier studies, reporting 80 percent (Reference Cram, Groves and Foster8) and 76 percent (Reference Menon and Marshall9), respectively.
However, our survey has several limitations. Our HB-HTA unit serves a single hospital and issued a small number of HB-HTA reports. In addition, it did not address the financial impact the decisions had on the hospital's budget, or the clinical impact they had on patients’ outcome. These two important issues should be incorporated into future follow-up studies.
CONCLUSION
Our survey at Lausanne University Hospital adds useful information from an additional country (Switzerland) to the few studies published over the effects and impact of local HB-HTA reports (two from Canada, one from France, and one from Austria), which was recently reviewed (Reference Gagnon, Desmartis, Poder and Witteman3). Its main message is that routine follow-up of HB-HTA reports is very important to inform the hospital management about the outcome of its decisions. HB-HTA was recently made more visible by the handbook issued by the European “AdHopHTA” project (Adopting Hospital-based Health Technology Assessment) (Reference Sampietro-Colom, Lach and Cicchetti2). This work should make it possible for every hospital to acquire the tools and skills to better assess the real value of new HTs. Only with this information can quality and safety of care be guaranteed, and reasonable investment decisions be taken.
CONFLICTS OF INTEREST
None declared.
