In the treatment of symptomatic severe aortic stenosis (AS), transcatheter aortic-valve implantation (TAVI) has become an essential alternative to surgical aortic-valve replacement (SAVR) for patients with severe AS who are either inoperable or at high risk for SAVR (Reference Smith, Leon, Mack, Miller, Moses and Svensson1–Reference Mack, Leon, Smith, Miller, Moses and Tuzcu3). This technique involves insertion, through a catheter, of a bioprosthetic valve, which is implanted within the patient's diseased aortic valve (Reference Smith, Leon, Mack, Miller, Moses and Svensson1). Through both progressive improvement in TAVI device design and increasing experience of interventional cardiologists, complications arising from TAVI have decreased (Reference Généreux, Cohen, Mack, Rodes-Cabau, Yadav and Xu4).
However, to achieve optimal procedural performance, interventional cardiologists using TAVI require training and accumulation of experience. Indeed, several studies have suggested that TAVI procedures are technically complex and have a significant learning curve (Reference Gurvitch, Tay, Wijesinghe, Ye, Nietlispach and Wood5–Reference Webb, Altwegg, Boone, Cheung, Ye and Lichtenstein8). In a trial report on a medical device by Motte et al., it appears essential to know how the learning curve was evaluated or how the training of operators was managed (Reference Motte, Diallo, van den Brink, Châteauvieux, Serrano and Naud9). A minimum amount of training for each operator is required to complete this learning curve, as well as carrying out a minimum number of procedures per year to maintain TAVI competency. There seemingly is no consensus on these minimum numbers, and no standardized guidelines when initiating a TAVI program currently exist. Some studies have stated that a minimum of twenty procedures is required to achieve a good level of practice (Reference Gurevich, John, Kelly, Raveendran, Helmer and Yannopoulos10). Training protocols are usually based on the participation of experienced proctors, dry laboratory sessions, or animal models (Reference Glauber, Murzi and Cerillo11). The French National Health Authority has suggested that carrying out two procedures per month was the minimum needed for each operator to maintain this technical competency (12;13).
With this in mind, medical device studies, including those regarding TAVI devices, require acknowledgment of a “roll-in” phase to account for the operator learning curve. A roll-in phase can be defined as a training phase, in which an operator uses an investigational medical device for an initial period on subjects enrolled in the study. The number of roll-in subjects is usually limited, and data are not included in the final analysis. In a trial report on TAVI, it clearly appears important to know how the training of the operators was managed. Indeed, it may be difficult to establish the external validity of the study, also called generalizability, without this information (Reference Motte, Diallo, van den Brink, Châteauvieux, Serrano and Naud9). In addition, if the roll-in phase is not conducted adequately, an insufficient training regime may lead to an increase in adverse events that are caused by operator error (Reference Amoore14). Thus, the quality of a clinical report, that is, providing information about the design, conduct, and analysis of the trial, is evidently separated from the methodological quality of the trial. To our knowledge, there are no studies on the quality of learning-curve reporting either in medical device studies in general or specifically on TAVI. In addition, as the expansion of indications for TAVI is now being suggested—for example in patients who are deemed to be at intermediate risk for surgery—it seems apt to study how information on the learning curve is reported in TAVI studies (Reference Reardon, Van Mieghem, Popma, Kleiman, Søndergaard and Mumtaz15). The aim of the present systematic review was to identify prospective studies on TAVI over the past 10 years, then to analyze the quality of information reported about the learning curve and finally to strengthen the health technology assessment of TAVI by improving the reporting of an essential aspect of device evaluation.
Materials and Methods
Study Selection
This systematic review was performed following the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines (Supplementary File 1). A study protocol was also established to clarify the review and to expose eligibility criteria (Supplementary File 2). The systematic search was performed using PubMed and EMBASE to collect studies regarding the learning curve in a TAVI context. The study protocol specifies the search terms used (Supplementary File 2). Limits were defined on publication date, language, and study design. The focus was narrowed to French or English prospective studies published between 2007 and 2017. We limited our search to 10 years, between 2007 and 2017, because TAVI devices were not launched on the market until 2007. Two independent reviewers screened titles and abstracts to exclude irrelevant or duplicate articles. Exclusion criteria of the articles identified were: not in English or French; not based on the source study; other study designs (i.e., not a prospective study); other subjects (not dealing with TAVI or not on TAVI alone); adverse event reporting; biomarker study; drug study; medical imaging study; or outcome monitoring study. Then, a full-text review was performed on the included articles with the same exclusion criteria as that applied to the title and abstract.
Data Analysis
Publications were first exported to Zotero V4.0.29.15 (2017), and as a second step, exported to Microsoft Office Excel® 2010. Then, a data extraction worksheet was developed in Microsoft Office Excel® 2010 to standardize data extraction and analysis. Various information was collected: first author, year of publication, country of origin, source study, study design (randomized/controlled study, comparative study, single-center, multicenter, and so on), brand name of TAVI, access type of TAVI, number of patients, author being a proctor, center type, and source of funding. To our knowledge, there are no international guidelines for the reporting of learning curve information in clinical studies on medical devices. Consequently, we developed our own checklist to assess the quality of information reported about the learning curve. This checklist is based on the literature on this topic. First, after a first reading of every article retrieved, we checked whether the term “learning curve” was mentioned in the study. Then, based on the work of Raman et al. (Reference Raman, Gaylor, Rao, Chan, Earley and Chang16), we reported where “learning curve” was mentioned in the study (introduction, methods, results, or discussion). The location in the text is an important element to consider. Indeed, if the learning curve is mentioned in methods or results, it is likely that the learning curve was anticipated and/or evaluated in the study. As stated by Raman et al., when the learning curve is mentioned only in the discussion, most authors described this point only as one of the factors influencing outcomes of the procedures, which is informative but rarely enough to fully appreciate the learning curve itself. We also considered in our analysis the linked articles (methodological references). Then, we searched for whether a proctorship was mentioned. We also allowed for other terms for “proctor” being used, such as “mentor” or “supervisor.” Arai et al. showed that an expert tutoring is likely to reduce the complication and mortality rates in patients undergoing transfemoral-TAVI owing to the effect on the operator's learning curve (Reference Arai, Lefèvre, Hovasse, Hayashida, Watanabe and O'Connor17). We checked whether a roll-in phase was mentioned (Reference Amoore14). As stated in the Introduction, this phase is essential to improve the operator's skills. When a roll-in phase was stated, we also searched the number of patients treated during this phase and/or whether training on animals was performed. Finally, we also searched for whether the number of patients to maintain skills was mentioned. This information is important to consider because it may greatly influence clinical outcomes when the technique is routinely used. The French National Health Authority recommends a minimum of two procedures per month to maintain skills with TAVI (13). The more a study included information for the learning curve checklist, the more informative it was considered. These data were reported into a Microsoft Office Excel® 2010 worksheet. Then, data from article characteristics and learning curve were combined for the analysis.
Results
Study Selection
After removing duplicates, 764 studies were identified, of which 693 were excluded on the basis of the content of their titles and abstracts. Seventy-one studies were considered in their entirety, following which a further fifteen were then excluded. We focused only on primary research articles, and twelve studies corresponding to methodological references from articles initially extracted were added to the review. Thus, a total of sixty-eight studies met the selection criteria and were suitable for complete analysis (Figure 1).
Figure 1. PRISMA flow chart of the included studies.
Characteristics of Included Studies
The details of the included sixty-eight studies are presented in Supplementary Table 1. There was a small increase in the number of studies published between 2006 (before 2007, for studies corresponding to methodological references from articles initially extracted) and 2009, which steadily increased between 2010 and 2012 and fell in 2013; numbers rose to previous levels in 2014 before declining again from 2015, with a little recovery in 2017 (Figure 2).
Figure 2. Number of publications per year.
Most of the studies (71 percent) were undertaken in five countries: Germany (n = 21; 31 percent), Italy (n = 10; 15 percent), U.S.A. (n = 9; 13 percent), France (n = 8; 12 percent), and Canada (n = 5; 7 percent) (Supplementary Table 2).
Of the sixty-eight studies, we only highlight seven (10 percent) randomized controlled trial studies, and thirty-nine (57 percent) comparative studies. The number of included patients in these studies ranged from 3 to 2,069 patients, and 33 (49 percent) of the studies recruited more than 100 patients. Thirty-four (50 percent) studies were single-center studies, and among these, twenty-nine (85 percent) were performed in a public center.
Data Synthesis
Learning curve data extracted from the studies included are presented in detail in Table 1. Moreover, thirty-seven of the included studies (54 percent) mention the learning curve. The learning curve is mentioned in the discussion for thirty-five (95 percent) of these studies; only three studies (8 percent) also mentioned the learning curve in the results section. In the three studies reporting the learning curve in the results section, general characteristics and outcomes are compared between initial patients and subsequent patients for assessing the learning curve (Reference Webb, Pasupati, Humphries, Thompson, Altwegg and Moss18–Reference Takagi, Latib, Al-Lamee, Mussardo, Montorfano and Maisano20). One study shows that the implantation rate is significantly different between both groups, that is, more favorable in the subsequent patients (Reference Hernández-Antolín, García, Sandoval, Almería, Cuadrado and Serrano19). Another study shows that the rates of procedural success, malposition, and intraprocedural mortality are more favorable, with statistical significance, in the subsequent patient group (Reference Webb, Pasupati, Humphries, Thompson, Altwegg and Moss18). The final study shows that valve malposition is more frequent in the initial patients; however, no statistical calculations were performed (Reference Takagi, Latib, Al-Lamee, Mussardo, Montorfano and Maisano20). In these three studies, the patient follow-up varies from 6 to 12 months.
Table 1. Learning curve data extracted from the included studies
N, no; NS, not stated; PMS, patient to maintain skills; Y, yes.
Moreover, we observed no trend between the year of publication and the reporting of learning-curve information. In addition, thirty-seven of the studies included (54 percent) do not report funding sources, and no association with the reporting of the learning curve was observed. Finally, sixty articles (88 percent) do not mention a roll-in period. Details of the eight studies mentioning a roll-in period are presented in Table 2.
Table 2. Details of the studies mentioning a roll-in period
NS, not stated; RIP, roll-in period.
In addition, it was noted that sixty-five of the studies (96 percent) do not disclose proctorship, whereas authors of twenty-five studies (37 percent) were proctors working with manufacturers of TAVI. Indeed, proctorship was not disclosed in the method section but in the competing interests. Also, we found no mention of the number of patients necessary to maintain skills in any of the studies included. Last, among randomized and controlled studies, three studies were found to mention learning curves (43 percent), and twenty-two studies alluded to learning curves among comparative studies (56 percent).
Discussion
To our knowledge, this study is the first systematic review dealing with the quality of information reported about the learning curve in TAVI procedures.
Most of the articles reviewed mention the learning curve (more than 50 percent), but only very limited reporting and detailing have been found on the learning curve itself (such as roll-in period, proctorship, and maintenance of skills). In spite of this, learning issues were often mentioned in discussions and were mostly considered to be a source of bias (25;26;30;31;33;37;40;55). Only a few studies provided outcome measures that showed the difference between patients treated in an early phase and those treated later in the study (Reference Webb, Pasupati, Humphries, Thompson, Altwegg and Moss18–Reference Takagi, Latib, Al-Lamee, Mussardo, Montorfano and Maisano20). In light of these studies, the implantation rate and the valve malposition were outcomes that directly connected to the training of the operators, and this is consistent with previous studies on the topic (Reference Lunardi, Pesarini, Zivelonghi, Piccoli, Geremia and Ariotti85).
In a few studies, the exclusion of a start-up phase is believed to be of paramount importance in providing a more objective evaluation of new devices, without skewing from less-experienced operators (Reference Khawaja, Wang, Pocock, Redwood and Thomas86). Moreover, the experience of the operators is not clearly stated in the studies; however, some authors specify that centers with more TAVI experience trended toward fewer complications than sites with little or no previous experience (Reference Greenbaum, Babaliaros, Chen, Stine, Rogers and O'Neill24;Reference Deeb, Reardon, Chetcuti, Patel, Grossman and Yakubov31;Reference Osnabrugge, Head, Genders, Van Mieghem, De Jaegere and van der Boon51;Reference John, Buellesfeld, Yuecel, Mueller, Latsios and Beucher57;Reference Kempfert, Van Linden, Kim, Blumenstein, Rolf and Möllmann70). These many allusions to the learning curve remain very heterogeneous in terms of details. As shown by Motte et al. (Reference Motte, Diallo, van den Brink, Châteauvieux, Serrano and Naud9), this is probably due to the fact that no guidelines exist for reporting clinical trials on implantable medical devices. However, this same article has determined some relevant items for reporting clinical trials on implantable medical devices, and the learning curve was identified as one of them.
It is remarkable to note that of studies that mention the learning curve, more than three quarters do not discuss a roll-in period. When a roll-in phase was described, it was very disparate between studies. Indeed, a roll-in patient number was not specified in two studies with a roll-in phase (Reference Takagi, Latib, Al-Lamee, Mussardo, Montorfano and Maisano20;Reference Leon, Smith, Mack, Miller, Moses and Svensson83). It was also difficult to know whether training sessions were carried out in humans or animal models in these articles. In addition, some studies described that training would end after a predetermined number of patients, at which point the operator could be considered well-trained. The studies proposed to compare the first patients for whom the technique has been used with the same number of patients after “expertise” is reached (Reference Webb, Pasupati, Humphries, Thompson, Altwegg and Moss18;Reference Kempfert, Rastan, Holzhey, Linke, Schuler and van Linden82). We remain skeptical about this predetermined roll-in patient number, which seems to have been always arbitrarily determined. Indeed, to our knowledge, although there is no consensus on a reasonable minimum number of patients, studies generally suggest that this number should be between ten and twenty cases (Reference Gurevich, John, Kelly, Raveendran, Helmer and Yannopoulos10;Reference Vallely, Wilson, Adams and Ng87). Consequently, most of the predetermined roll-in patient numbers we found here did not match this range of cases, and it is difficult to understand how the numbers were selected by the study designers.
With very few exceptions, the proctorship was not mentioned in the studies included. This must be linked with the notable percentage of authors who were proctors themselves. Many authors argue that acquisition of expertise in technology and procedures, such as TAVI, must be a gradual process based on an educational experience that should pass through a structured training with different levels of supervision, such as the preceptorship and the proctorship (Reference Gurevich, John, Kelly, Raveendran, Helmer and Yannopoulos10;Reference Glauber, Murzi and Cerillo11). This experiential training is important to consider, partly because the number of TAVI programs is proliferating and that it is necessary to plan learning phases in these future programs. It has also been demonstrated that a careful monitoring of the learning curve may have important clinical and economic implications in the development of TAVI procedures (Reference Alli, Rihal, Suri, Greason, Waksman and Minha7;Reference Lunardi, Pesarini, Zivelonghi, Piccoli, Geremia and Ariotti85). Although evolving skills can be evaluated during the supervision period, the learning curve goes on after completing any form of training and is a lifelong process. To monitor a center's TAVI outcomes or individuals performing the procedure, some statistical tools can be used. The cumulative summation (CUSUM) test was first introduced to analyze time-series based on industrial processes, and most recently the learning curve-cumulative summation (LC-CUSUM) was designed to determine when a level of expertise has been attained (Reference Arai, Lefèvre, Hovasse, Hayashida, Watanabe and O'Connor17;Reference Papanna, Biau, Mann, Johnson and Moise88). This method has been reported to be useful in monitoring a learning curve with regard to the incidence of perioperative complications. Therefore, studies dealing with TAVI procedures should ideally evoke a protocol in which a learning curve would have been anticipated, but we found that in many studies this was not the case (Reference Motte, Diallo, van den Brink, Châteauvieux, Serrano and Naud9).
The introduction of new-generation percutaneous aortic valves in clinical practice obliges operators to undergo continuous technical updating and investigate possible limitations of the newly adopted devices (Reference D'Ancona, Agma, Ince, El-Achkar, Dißmann and Ortak27;Reference Modine, Sudre, Delhaye, Fayad, Lemesle and Collet71). The expertise of operators could help to improve not just the practices, but also the existing devices. Then, the evolution of the devices must progress toward that of an ease of use and a rapid learning. Furthermore, various transcatheter aortic valves exist on the market, and hospitals can sometimes be forced to change their percutaneous valve preference because of cost considerations. This requires further training with these new devices.
In light of our present work, we propose some criteria that may be helpful when considering the generalizability of results from medical device studies regarding the learning curve. First, the roll-in phase should be systematically reported in the methods section, and the number of patients treated in the roll-in and the duration of this phase should be specified. This could be presented as a protocol associated with the article explaining this in detail. Also, the involvement of a proctor should be reported because it has been proved that expert tutoring provides better results to improve patient outcomes (Reference Arai, Lefèvre, Hovasse, Hayashida, Watanabe and O'Connor17). Finally, if no specific training has been necessary because the operators are already well-trained with the technique used in the paper (e.g., when the technique is not new), this should be explained and justified by providing the experience of the operators. It could be helpful to know, for example, the number of cases already treated with the technique by each operator.
The present study has several limitations that should be addressed. First, we did not retrieve full-text versions of all the articles we identified. Although we made every effort to collect the articles, some articles were unobtainable. In addition, we only focused here on articles referenced by PubMed and EMBASE. We did not used additional databases for this systematic review due to resource and time restrictions.
Conclusion
The present systematic review highlights that many prospective studies on TAVI over the past 10 years mention learning curve as a core component of successful TAVI procedures. However, the quality of information about the learning curve reported is relatively poor; only very limited reporting material has been found, whereas some details about roll-in period, proctorship, number of cases to maintain skills should be expected.
Supplementary Material
The supplementary material for this article can be found at https://doi.org/10.1017/S0266462320000100
Conflict of Interest
The authors declare that they have nothing to disclose.
