In most cases, the implementation of innovation in health care requires a learning period, characterized by a progressive increase in productivity, that is, a decrease of resource consumption and/or increased health outcomes. This improvement in performance over time is currently represented by the learning curve of innovation (Reference Cook, Ramsay and Fayers4;Reference Ramsay, Grant and Wallace10).
Because of these potentially important increases in productivity, learning effects may have huge consequences on the results of cost effectiveness analyses. Ignoring learning effects may lead to a highly increased incremental cost-effectiveness ratio of a new technique, especially when economic evaluation is performed at an early stage of innovation. Consequently, learning effects should be carefully analyzed. Nevertheless, they are often underinvestigated, as reported in the literature review by Ramsay et al. (Reference Ramsay, Grant and Wallace10). These authors have identified 272 studies exploring surgical (92 percent) or diagnostic (8 percent) innovations. In most cases, the variable used as a proxy for learning is the time spent to implement the innovative procedure. However, quantitative analysis is generally rather poor given that most studies use a split group method but do not provide a rationale for determining the cut point for allocating patients to the learning or the routine group according to the experience of professionals (either individuals or teams). Moreover, statistical analyses are generally limited to univariate tests. When multivariate analysis is performed, professional experience is reported as a dichotomous variable and few studies have investigated interoperator variations. Ramsay et al. (Reference Ramsay, Grant and Wallace10) have also searched for relevant statistical methods used to analyze learning effects in other fields than health care. Their conclusion was that multilevel analyses should be preferred because they take the hierarchical structure of data, and thereby interoperator variability, into account. Such methods have already been applied to surgery (Reference Cook, Ramsay and Fayers3).
As far as we know, only one other study has used a multilevel method to analyze learning effects in the cost evaluation of a health technology (Reference Bonastre, Noel and Chevalier2). The authors were interested in radiation treatments using intensity modulated radiation therapy (IMRT), and they have shown that the total cost per patient in a hospital setting decreases as more and more patients are treated.
Indeed, learning effects are particularly important to consider in radiotherapy. First, radiotherapy, which is a common treatment for many tumors, accounted for 10 percent of the total cost of cancer care in France in 2004 (Reference Amalric1). Second, modern radiotherapy involves costly equipments and technological innovation has been very fast in recent years. However, the analysis of learning effects may be complex in radiotherapy, because it is sequential and involves many sessions for the same patient. Thus, learning effects can come from both (i) intrapatient learning, that is, learning from one session to the next for a given patient, and (ii) interpatient learning, that is, learning from one patient to the next in a given setting. It may then be difficult to distinguish between these two learning mechanisms.
Our study aimed at analyzing learning effects associated with a new radiotherapy technique, using multilevel analysis to separate intrapatient learning from interpatient learning. Because of the respective positions of patients and health professionals, intrapatient learning could be due to both the patients and the professionals, whereas interpatient learning only derives from professional learning. As a consequence, we wanted to know whether intrapatient and interpatient learnings were related and whether they might be associated with some patient or professional characteristics. Moreover, for further comparison of the new technique to conventional radiotherapy, we wanted to know to which extent the learning phenomenon could be attributable to innovation.
METHODS
Data Set
The new radiotherapy technique under investigation is respiration gated radiotherapy (RGR) using breath-hold techniques, which aims at improving the delivery of radiation to tumors affected by respiratory motion such as lung and breast tumors. A national prospective observational multicenter study supported by the French Ministry of Health was conducted from January 2004 to December 2005. In seven of the participating hospitals, this study aimed at analyzing both health effects and costs of RGR as compared to conventional radiotherapy. By increasing the conformality of radiation delivery, RGR should decrease complication rates in organs at risk (lung, heart, etc.) (Reference Giraud, Reboul and Clippe5;Reference Hanley, Debois and Mah6;Reference Ohara, Okumura and Akisada8;Reference Pedersen, Korreman, Nyström and Specht9;Reference Remouchamps, Vicini and Sharpe11–Reference Simon, Giraud and Dumas13). However, costs would be higher, due to extra working time of medical staff and medical technicians on the one hand and extra use of equipments such as linear accelerators on the other hand.
As mentioned above, this study aims at analyzing the learning effects of the new radiotherapy technique, then at specifying whether they are attributable to innovation by comparison with conventional radiotherapy. Indeed, these might be major issues to consider for performing economic evaluation at an early stage of innovation.
In each of the seven hospitals, the medical staff and medical technicians involved in the delivery of radiotherapy were considered together. Indeed, although all radiotherapy units were quite big (six radiotherapists or more), only one or two radiotherapists used the new technique, and these two worked in close collaboration. At the beginning of the study, experience with the new technique in each hospital was measured by the number of patients previously treated by RGR. Moreover, radiotherapists in each setting were asked how many patients they considered should be treated with RGR (starting with the first one treated in their hospital) before they acquire expertise in using the technique, that is, before it becomes routine. Both data are reported in Table 1.
Table 1. Experience with the New Technology and Opinion of Radiotherapists at the Beginning of the Study in Each of the Seven Participating Hospitals
RGR, respiration gated radiotherapy.
For each patient, radiation treatment involves two stages: preparation and delivery, that is, radiation sessions. For the estimation of resource consumption, preparation is far less important than delivery itself, because treatment delivery requires a great number of homogeneous sessions in which learning effects may play a major role. For these reasons, we chose to focus our analysis on treatment delivery.
Patients were enrolled in seven settings, two university hospitals, and five comprehensive cancer centers. With regard to the innovative technique, 3,908 radiotherapy sessions were considered, corresponding to 132 patients with lung cancer (n = 96) or breast cancer (n = 23). Up to 40 sessions were administered to each lung cancer patient and 25 to each breast cancer patient. With regard to conventional radiotherapy, 4803 sessions were taken into account, corresponding to 170 patients with lung cancer (n = 81) or breast cancer (n = 89).
Each session duration was collected, from the time of patient's entry into the treatment room to his/her leaving the room. Apart from session duration, the data collected for each session were the type of disease, the patient's enrollment number (in the hospital where the treatment was delivered), and of course the setting where the session was administered. The dose delivered at each session was not considered, because it did not vary, either across sessions for the same patient or across patients.
Multilevel Modeling
Given the longitudinal nature of the data, statistical analysis applied to innovation was performed using an individual growth model (Reference Hox7;Reference Singer and Willett14). In its simplest form, the multilevel model for repeated data leads to a two level-model, with the measurement occasions (here, sessions) as level-1 units and individuals (here, patients) as level-2 units. Within individual change over time is then modeled at level 1, whereas interindividual differences are modeled at level 2. Considering hospitals as level-3 units would have been appropriate, because patients were nested in hospitals. However, the key condition for introducing a further level is that related units can be considered as a random sample from a wider population. In practice, at least twenty units are needed, which was not the case here because data were collected from only seven hospitals.
Level 1 Model. The individual growth trajectory, defined as the evolution of session duration for each patient, was modeled as a function of the session number. Graphic representation of individual trajectories suggested that duration actually decreased with the number of sessions, but less and less over time. As a consequence, the level-1 model was estimated using a logarithmic transformation on session duration, which improved the quality of the adjustment. A possible decrease in session duration for a given patient refers to the intrapatient learning effect.
More precisely, the dependent variable was the duration of session i for patient j, where i varied from 1 to 40 for lung cancer patients and from 1 to 25 for breast cancer patients. The level-1 equation could be written as follows: log Yij = αj + ßjNij + ɛij, where Yij is the duration of session i for patient j and Nij = i−1, so that Nij = 0 for the first session. The initial status is αj, because αj = log Y1j where Y1j is the duration of the first session for patient j. The rate of change of session duration (from one session to the next) is constant and equal to exp (ßj)−1, very close to ßj if the rate is low, and ɛij ~ N (0, σɛ2) is the error term.
Level 2 Model. To investigate the effect of professional experience associated with the new radiotherapy technique, the rank order of patients treated with the new technique in a given hospital was introduced at level 2. Indeed, this variable, labeled EXPERj, varies across patients, but keeps the same value for all sessions of a given patient.
On the one hand, EXPERj can influence initial status, that is, the duration of the first session for a given patient in a given setting, which can then decrease from this patient to the next one. This possible decrease in the first session duration will be labeled as the first-type interpatient learning effect. On the other hand, EXPERj can also impact the rate of change of session duration for a given patient, which will be labeled as the second-type interpatient learning effect.
In the level 2 equation related to initial status, the graphic representation of αj as a function of EXPERj suggested a decrease that was less and less marked as experience increased. Consequently, a possible effect of EXPERj was tested using its logarithm. Moreover, we hypothesized that initial status could be influenced by one of the most important patient characteristics, that is, type of disease labeled DISj: according to radiotherapists, RGR treatment was more complex for lung cancer than for breast cancer, thus longer session durations were required.
As mentioned above, hospital characteristics could not be considered at third level in the model. However, hospital status could entail differences in the management of the radiotherapy department, thereby in the learning phenomenon. Thus we introduced hospital status (labeled HS) at level 2 as an environmental variable to obtain appropriate model parameter estimates.
The level-2 equation related to initial status could be written as follows: αj = α0+ α1 log EXPERj + α2 DISj + α3 HS + uj, where DISj is coded 1 for lung cancer and 0 for breast cancer and HS is coded 1 for university hospitals and 0 for comprehensive cancer centers. The α0, α1, α2, and α3 are the estimated equation parameters.
In the level-2 equation related to the rate of change, graphic representation of ßj as a function of EXPERj did not suggest any relationship between the two factors. However, a possible effect of professional experience was tested by introducing EXPERj in the equation. As done previously, and for the same reasons, the type of disease and hospital status were also introduced.
The level 2 equation related to the rate of change could then be written as follows: ßj = ß0+ ß1EXPERj + ß2 DISj + ß3 HS + vj, where ß0, ß1, ß2 and ß3 are the estimated equation parameters.
The term errors of the two level-2 equations are such that:
![\begin{equation}
\left( \begin{array}{@{}c@{}}
\displaystyle{u_j} \\[6pt]
\displaystyle{v_j} \\
\end{array} \right) \to N
\left( \left[\displaystyle \begin{array}{@{}c@{}}
0 \\[6pt]
0 \\
\end{array} \right],\left[\displaystyle \begin{array}{@{}c@{\quad}c@{}}
{\sigma_u^2} & {\sigma_{uv}} \\[6pt]
{\sigma_{uv}} & {\sigma_v ^2} \\
\end{array} \right] \right)\end{equation}](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:20160921054921559-0955:S0266462309090291:S0266462309090291_eqnU1.gif?pub-status=live){kind=small}
RESULTS
Learning Effects of the New Technique
Session durations observed with the new technique in each hospital and for each disease are reported in Table 2. As an overall result, session duration was 18.3 minutes in average, with a 0.5 variation.
Table 2. Session Durations in the Seven Participating Hospitals, According to Disease (Minutes)
aNumber of sessions observed during the study period.
Results of the multilevel model and the unconditional mean model are presented in Table 3. The unconditional mean model, with no predictors at either level, gives the partitioning of the variability of session duration within and between persons. This model indicated that half (52.7 percent) of the total variation in session duration was attributable to differences across patients, which justifies multilevel modeling.
Table 3. Results of Modeling for Innovation
Note. Initial status: duration of the first session for patient j (logarithm); rate of change: percentage of variation in session duration from one session to the next for patient j; EXPER: professional experience (number of patients previously treated with respiration gated radiotherapy); DIS: type of disease (lung or breast cancer); HS: hospital status (university hospital or comprehensive cancer center).
*p < .05; **p < .01; ***p < .001.
As expected, a first-type interpatient learning effect was highlighted: the duration of the first session significantly decreased when professional experience improved (Table 3). However, this duration depended on the type of disease, because it was longer for patients with lung cancer than for those with breast cancer. With regard to hospital status, the duration of the first session was longer for patients treated in university hospitals than for those treated in comprehensive cancer centers.
An intrapatient learning effect was also estimated: the rate of change of session duration for a given patient was significantly different from 0 and negative (Table 3). However, it was not significantly influenced by professional experience, which means that there was no significant type 2 interpatient learning effect. Again, the rate of change of session duration did depend on the disease.
Estimates of Session Duration on Prototypical Patients and Focus on Lung Cancer Patients Treated in a Comprehensive Cancer Center
As an overall result, session duration estimates show considerable variation. For example, the estimated duration of the last session for the 100th breast cancer patient treated in a comprehensive cancer center is approximately 12 minutes, whereas the estimated duration of the first session for the first lung cancer patient treated in a university hospital is close to 28 minutes, which is more than twice longer.
The first-type interpatient learning effect, measured as the decrease of the first session duration from the first patient to the 100th, was estimated at 14 percent, whatever the disease. The duration of the first session itself depended on the type of disease, with a 19 percent increase of the estimated value between breast cancer and lung cancer.
Considering the intrapatient learning effect, measured as the decrease in duration from the first to the last session for a given patient, it depended on disease type and was estimated at 19 percent and 26 percent for lung cancer (last session: 40th) and breast cancer (last session: 25th), respectively. This intrapatient learning effect was also found with conventional radiotherapy, without any significant modification due to innovation.
As an example, let us consider the case of prototypical lung cancer patients treated in a comprehensive cancer center. Estimates of session duration are reported in Table 4 as a function of professional experience and of session number for a given patient.
Table 4. Estimated Session Duration as a Function of Professional Experience and Session Number for Prototypical Lung Cancer Patients Treated in a Comprehensive Cancer Center (Minutes)
a Fortieth session.
Evolution of First Session Duration Estimates: First-Type Interpatient Learning Effect. Our results demonstrate a substantial first-type interpatient learning effect, with a 14 percent decrease in the first session duration between the 1st patient (23.3 minutes) and the 100th patient treated (20.1 minutes). The estimated duration of the first session as a function of professional experience is represented in Figure 1. This is the first-type interpatient learning curve in this situation.
Figure 1. First-type interpatient learning curve: estimated duration of the first session as a function of professional experience for prototypical lung cancer patients treated in a comprehensive cancer center (minutes).
According to our model specification, the first-type interpatient learning curve strongly decreased when professional experience was low. As an example, the estimated average decrease rate from one patient to the next was 0.5 percent from the 1st to the 20th patient and only 0.05 percent from the 50th to the 100th (Table 4; Figure 1). Of course, the decrease became extremely slow after the 100th patient.
Radiotherapists' estimations are recalled in Figure 1. All their expectations seemed rather optimistic at the beginning of the study, especially when the radiotherapists had little experience but also, and more surprisingly, when experienced radiotherapists were involved.
Evolution of Session Duration Estimates for a Given Patient: Intrapatient Learning Effect and Second-Type Interpatient Learning Effect. As previously outlined, the rate of change of session duration for a given patient was significantly negative, but was not significantly influenced by professional experience (Table 3). Indeed, it only depended on the disease.
Again, let us consider the case of prototypical lung cancer patients treated in a comprehensive cancer center. There was an estimated 19 percent decrease in session duration from the first to the last session for a given patient, independently of professional experience (Tables 3 and 4).
In Figure 2, the estimated session duration was defined as a function of the number of sessions for these patients, according to professional experience. The four corresponding prototypical trajectories, corresponding to the 1st, 20th, 50th, and 100th patients treated, respectively, represent the intrapatient learning curves; the only difference came from the duration of the first session because the rate of change was identical.
Figure 2 Intrapatient learning curves: estimated session duration as a function of the session number for prototypical lung cancer patients treated in a comprehensive cancer center, according to professional experience (minutes).
Learning Effects Attributable to Innovation
As mentioned above, learning effects should be taken into account for comparing the new technique to conventional radiotherapy. Knowing that conventional radiotherapy had been implemented in all seven hospitals for many years, interpatient learning was not at work anymore. However, an intrapatient learning effect, that is, a possible decrease in session duration for a given patient, could be present both with conventional radiotherapy and with the new technique.
The mean session duration for the 4,803 sessions of conventional radiotherapy was 12.3 minutes, with 0.3 variation. Statistical analysis used the same multilevel model as before, except that professional experience was not taken into account. Results of the multilevel model applied to conventional radiotherapy are presented in Table 5.
Table 5. Results of Multilevel Modeling for Conventional Radiotherapy
Note. Initial status: duration of the first session for patient j (logarithm); rate of change: percentage of variation in session duration from one session to the next for patient j; DIS: type of disease (lung or breast cancer); HS: hospital status (university hospital or comprehensive cancer center).
*p < .05; **p < .01; ***p < .001.
As was observed in the case of innovation, the first session duration depended on the type of disease; it was longer for lung cancer than for breast cancer (Table 5). Likewise, the duration of the first session was longer for patients treated in university hospitals than for those treated in comprehensive cancer centers.
Of interest, an intrapatient learning effect was also at work, because the rate of change of session duration over time for a given patient was significantly negative (ß0 parameter). Again, session duration decreased more quickly for breast cancer than for lung cancer (ß2 parameter). Statistical comparison of ß0 estimates between innovation and conventional radiotherapy (Tables 3 and 5) shows that they are not significantly different at the 5 percent level. The same stands true for ß2 estimates (Tables 3 and 5). As a consequence, intrapatient learning effects did play a role when using conventional radiotherapy and were not significantly modified by innovation. Thus, only interpatient learning was specific to innovation, that is, type-1 interpatient learning given that there was no significant type 2 interpatient learning effect.
DISCUSSION AND CONCLUSIONS
Findings
To analyze the implementation of a new radiotherapy technique at an early stage, we used a multilevel analysis, which Ramsay et al. (Reference Ramsay, Grant and Wallace10) considered more appropriate for analyzing learning effects. Indeed, in so doing, we were able to identify different types of learning effects, namely an intrapatient learning effect and two interpatient learning effects, of the first and second types, respectively.
A first-type interpatient learning effect was measured by a significant decrease of the duration of the first radiotherapy session from one patient to the next in a given setting. Of interest, but not surprisingly, this duration was longer for lung cancer than for breast cancer due to higher complexity. As expected, the decrease rate of the first session duration was not stable but progressively declined over time, when professional experience improved.
An intrapatient learning effect was also identified; it corresponded to a significant decrease in session duration from one session to the next for a given patient. This decrease, modeled as a constant rate of change, was influenced by disease type; it was lower for lung cancer than for breast cancer, due again to higher irreducible treatment complexity.
For interpreting intrapatient learning, one must bear in mind that it likely resulted from a complex process related to both the patient and the professionals. Moreover, quality controls were systematically performed, especially at the beginning of the treatment. This could result in a higher rate of change during the first sessions, but we were not able to test this hypothesis because our rate of change was modeled as a constant throughout the treatment period.
Most interestingly, we did not find any significant second-type interpatient learning effect: the rate of change of individual trajectories was not affected by professional experience. Moreover, intrapatient learning effects were also identified when using conventional radiotherapy, and they were not significantly modified by innovation. As a consequence, in this situation, the intrapatient learning effect does not seem to be attributable to innovation, but rather to the sequential aspect of radiotherapy treatment. This point, however, would need further and more detailed investigation.
Literature on Learning Effects in the Health Care Field
Considering first-type interpatient learning, we clearly identified learning curves of the standard type, with increasing improvement in performance over time due to accumulation of professional experience. Our results are consistent with those from other authors (Reference Bonastre, Noel and Chevalier2;Reference Cook, Ramsay and Fayers4;Reference Ramsay, Grant and Wallace10).
Moreover, we clearly highlighted that some intrapatient learning was at work but that there was no significant second-type interpatient learning effect. To our knowledge, these different learning effects had never been simultaneously investigated and measured in the healthcare field.
Limitations of Our Study
Even if we did our best to fit observational data using multilevel modeling, reality is much more complex than any model can describe. As an example, the rate of change of an individual trajectory was modeled as a constant for a given patient, which might not be the case.
Second, if we obtained very simple findings, it has something to do with the restricted number of control variables used. The statistical significance of variance components at each level indicates that some intra- and interpatient variations remain unexplained, due to unobservable patient demographic or clinical characteristics, thus introducing a bias in individual parameter estimates.
Moreover, adding a third hospital level to our statistical analysis was not feasible in the framework of the present study. Consequently, potentially important characteristics of hospitals, such as the type of management of the radiotherapy department and the degree of disease or technical specialization of the staff, could not be investigated.
Impact of Our Results on Resource Allocation
The type of disease affects session duration. As a consequence, resource allocation could be modulated according to the disease. Regarding learning effects, it is important to make a clear distinction between intrapatient and interpatient learnings. This is particularly true in the framework of cost-effectiveness analyses performed for prospective purposes to facilitate decision making when deciding on the dissemination, or not, of new techniques. In our study, the first-type interpatient learning effect was responsible for an estimated 14 percent decrease of session duration (first session, from the 1st to the 100th patient treated). Likewise, the intrapatient learning effect was measured as a decrease in session duration (from first to last, for a given patient) estimated at 19 percent and 26 percent in lung and breast cancers, respectively. However, there was no significant second-type interpatient learning, which means that intrapatient learning did not change over time. Moreover, intrapatient learning effects were also present when using conventional radiotherapy and they were not significantly modified by innovation. Therefore, in our case, the first-type interpatient learning effect is the only one that should be considered to anticipate a decrease in the incremental cost-effectiveness ratio of the innovative technique in the future. However, should intrapatient learning be accelerated by professional experience, both types of interpatient learning effects should be considered.
How to Take Learning Effects into Account?
As highlighted above, learning effects may have considerable influence on performance, thereby on cost-effectiveness and ultimately on resource allocation. It is worth emphasizing here that professional expertise on learning effects should be considered with much caution. The radiotherapists questioned at the beginning of the study on the number of patients who should be treated with the new technique before it becomes routine (from the first one treated in their setting) provided very heterogeneous answers. They were also very optimistic, even in centers with extensive experience at the beginning of the study. Indeed, a comparison of their answers to the first-type interpatient learning curves (Figure 2) seems to indicate that they highly underestimated the learning phenomenon. However, these answers may also reflect different conceptions of routine. Consequently, given the potential importance of learning effects, careful analysis on observational data seems unavoidable.
CONTACT INFORMATION
Magali Morelle, MSc (morelle@lyon.fnclcc.fr), Statistician, Raphaël Remonnay, MSc (remonnay@lyon.fnclcc.fr), Economist, GATE, CNRS-UMR 5824, Health Economics Unit, University of Lyon, Centre Léon Bérard, 28, rue Laennec, Lyon, France 69008
Philippe Giraud, MD, PhD (philippe.giraud@egp.aphp.fr), Professor, Radiotherapy Department, Hôpital Européen Georges Pompidou, 20–40 rue Leblanc, 75015 Paris, France; Professor, University Paris V, 12, rue de l'École de Médecir, Paris, France 75006
Marie-Odile Carrère, PhD (carrere@lyon.fnclcc.fr), Professor, GATE, CNRS-UMR 5824, Health Economics Unit, University of Lyon, Centre Léon Bérard, 28, rue Laennec, Lyon, France 69003
