In cost-effectiveness studies performed alongside clinical trials, healthcare utilization can be measured using questionnaires or diaries completed by the patients in the trial or obtained from medical, billing, or other administrative records. The latter is often regarded as more accurate than the first. However, retrieving data from medical or administrative records can be time consuming and costly, especially when patients contact many different care providers. Furthermore, data on services for which the patient pays out-of-pocket, such as over-the-counter medication or alternative therapists, are missed using medical or administrative records only. Self-reported surveys, such as questionnaires or diaries, can provide data on all types of healthcare utilization, but can be less valid due to recall-bias. Several studies compared self-reported healthcare utilization with data from medical records, but results are inconclusive. Some studies found good agreement between both sources (Reference Marks, Lee and Slezak9;Reference Reijneveld13;Reference Van Den Brink, Van Den Hout, Stiggelbout, Velde and Kievit18;Reference Weissman, Levin and Chasan-Taber20), but others reported substantial differences (Reference Bellon, Lardelli, Luna and Delgado2;Reference Cronan and Walen4;Reference Goossens, Rutten-van Molken, Vlaeyen and Van Der Linden6). In general, agreement seems fairly good on major events such as hospitalizations or visits to the emergency department, but self-report of outpatient visits, visits to the general practitioner, and diagnostic, laboratory or imaging procedures seems less valid compared with medical records (Reference Bhandari and Wagner1;Reference Lubeck and Hubert8;Reference Petrou, Murray, Cooper and Davidson12;Reference Roberts, Bergstralh, Schmidt and Jacobsen14;Reference Ungar and Coyte17. Although several studies reported about the extent of agreement between self-reported healthcare utilization and data from medical records (Reference Bhandari and Wagner1;Reference Cronan and Walen4;Reference Lubeck and Hubert8;Reference Petrou, Murray, Cooper and Davidson12;Reference Van Den Brink, Van Den Hout, Stiggelbout, Velde and Kievit18, the impact of the different types of data collection on cost(-effectiveness) has not been studied. This is an important issue, because almost perfect agreement in hospitalizations between two data sources can still result in a substantial difference in costs as a result of the high costs of an inpatient day. On the other hand, a substantial difference in visits to the general practitioner may have little impact on costs, because of its low unit costs. The aim of this study was to compare the impact of using either self-reported resource use or resource use as obtained from administrative data of healthcare providers on costs and cost-effectiveness in a sample of patients with chronic obstructive pulmonary disease (COPD). Furthermore, we explored whether differences in cost estimates between the two different types of data sources were related to patient characteristics.
METHODS
Design of the Trial, the Intervention, and the Cost-effectiveness Study
Data were obtained as part of a cost-effectiveness study performed alongside a 2-year randomized controlled trial evaluating the effect of an INTERdisciplinary COMmunity-based COPD management program (INTERCOM) (Reference Hoogendoorn, van Wetering, Schols and Rutten-van Molken7). The trial included patients with COPD and impaired exercise performance who were recruited from two general hospitals in the Netherlands. One-hundred ninety-nine patients were randomized to the INTERCOM program (n = 102) or usual care (n = 97).
The INTERCOM program consisted of exercise training, education, and smoking cessation support offered by local physiotherapists in the proximity of the patient's home and by respiratory nurses in the hospital. Nutritionally depleted patients in the INTERCOM group were referred to a local dietician for counseling and nutritional supplements (Respifor®). The program was divided in a 4-month intensive intervention phase followed by a 20-month maintenance phase. During the intensive intervention phase, all patients visited the physiotherapist twice a week, the respiratory nurse on average two times and the dietician four times if they were nutritionally depleted. In the maintenance phase, these frequencies were lower: once a month for the physiotherapist and at 6, 9, 12, and 24 months for the dietician. Visits to the respiratory nurse during the maintenance phase were upon request and varied widely between patients from 0 to 16 visits. Patients assigned to usual care received pharmacotherapy according to accepted guidelines, a short smoking cessation advice by their respiratory physician and short nutritional advice to eat more and better in case they were nutritionally depleted. Quality of life and several functional parameters were assessed at baseline, 4, 12, and 24 months. All patients gave written informed consent.
The cost-effectiveness study was conducted from a societal perspective including all COPD and non-COPD-related healthcare costs, travel expenses, and costs of productivity losses. All costs related to conducting the trial have been excluded. Health outcomes were expressed in terms of quality-adjusted life-years gained (QALYs), using EQ-5D utility values.
Self-report Versus Care Provider Registrations of Resource Use
During the whole 2-year study period, healthcare utilization was recorded weekly in a cost booklet. In this booklet, patients recorded visits to general practitioners, medical specialists, physiotherapists, dieticians, respiratory nurses, alternative therapists, psychologists, social workers, use of over-the-counter medication and medical devices, hospital admissions, ambulance rides, time lost from paid work, hours of (un)paid household help, number of units of Respifor® used and use of other nutritional supplements. For all visits to care providers, the travel distance was recorded to be able to calculate travel expenses. Each booklet covered a period of 4 weeks and was collected every 2 months. In case the recorded information was unclear, patients were contacted by the investigators by telephone for further clarification.
Next to the self-reported data from the cost booklet, resource use was obtained from administrative data of different care providers. Information on the delivery and costs of outpatient medication was obtained from the patients’ local pharmacies. For twelve patients using oxygen, the start and stop dates of oxygen supply were obtained from their oxygen supplier. The number of hospitalizations, inpatient hospital days, and outpatient visits to medical specialists were obtained from the administrative systems of the two hospitals in the study. All seventeen local physiotherapists who treated patients in the INTERCOM group provided information about the number of contacts, the date, duration of the visits, and whether treatment was for the INTERCOM study or not. The six respiratory nurses involved in the study provided the same information for outpatient visits to the respiratory nurses for all patients in the INTERCOM group. The five local dieticians who treated nutritionally depleted patients in the INTERCOM group provided detailed information about the visits to the dietician. Finally, the number of units of Respifor® supplied to all nutritionally depleted patients in the treatment group was obtained from the supplier (Nutricia Netherlands).
Resource utilization was valued using Dutch guideline prices updated to the year 2007 (Reference Oostenbrink, Bouwmans, Koopmanschap and Rutten11). More details about the cost calculation and the cost per unit used can be found elsewhere (Reference Hoogendoorn, van Wetering, Schols and Rutten-van Molken7), but the most important unit costs are summarized in the online appendix (Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc).
Two Different Estimates of Cost-Utility
Cost-utility was calculated in two different ways. In the first analysis, data on healthcare utilization were based entirely on self-reported data from the cost booklet. Only data on outpatient medication and oxygen use were obtained from registrations as no self-reported data were available.
In the second analysis, data on healthcare utilization were based on registrations. This implied that outpatient medication, oxygen use, hospitalizations and visits to the medical specialist in the two hospitals in the study, visits to local physiotherapists and respiratory nurses in the hospital and visits to local dieticians and units of Respifor® used were based on registrations. The travel expenses for visits obtained from the registrations were calculated based on the average distance to the healthcare provider (hospital: 7.0, local physiotherapist: 1.8 and local dietician: 3.9 kilometers) (Reference Oostenbrink, Bouwmans, Koopmanschap and Rutten11). Data on visits to other care providers, use of over-the-counter medication and medical devices, ambulance rides, time lost from paid work, hours of (un)paid household help, travel expenses for visits to other care providers and use of other nutritional supplements besides Respifor® were based on the cost booklet, because data from registrations were not available for these data sources.
Statistical Analyses
All randomized patients who had at least one outcome measurement after start of treatment and completed at least one cost booklet were included in this study. Missing data could be the result of drop-out or unavailability of registrations or cost booklets while patients were (still) in the trial. The percentage of missing data for the different data sources was calculated as the total number of weeks with missing data summed over all patients divided by the maximum number of observable weeks if all patients had complete data for the entire 2-year study period (=18200).
Correlation between resource use data from the registration and self-reported resource use from the cost booklet was calculated using Spearman's rank correlation coefficient (rho). Furthermore, the proportion of perfect agreement between the two data sources was determined, where perfect agreement was defined as no difference between the two data sources. The correlation between the registrations and the cost booklet was calculated for the whole 2-year period, but also for 0 to 4 months, 4 to 12 months, and 12 to 24 months to see whether correlation changes over time.
After valuation of resource use, the absolute difference in total costs was calculated for each patient as the total costs based on the registrations minus the total costs based on the cost booklet. Multivariate linear regression analysis with the absolute difference in costs as depend variable was performed to investigate whether treatment group, drop-out, gender, age, number of co-morbidities at baseline, disease severity, health status, and total costs were associated with either under- or overreporting. Underreporting was defined as higher costs based on the registrations compared with the cost booklet, while overreporting was defined as higher costs as obtained from the cost booklet compared with the registrations. In these analyses, data from patients who did not complete the full 2 years of the trial were included in the analyses up until the moment patients dropped out and no imputation of missing data was done.
To account for costs and health outcomes that were missing due to drop-out and the additional uncertainty that these missing values introduce, “multiple imputation” was applied before calculating the cost-utility (Reference Rubin and Schenker16). This was done separately for missing costs based on self-reported resource use from the cost booklet and missing costs based on resource use from registrations. Each missing value was replaced by ten simulated values using the propensity score method in SAS V8 (Reference O'Callaghan10;Reference Rosenbaum15). Missing EQ-5D scores were imputed simultaneously with costs. More details about the multiple imputation are described elsewhere (Reference Hoogendoorn, van Wetering, Schols and Rutten-van Molken7).
Each of the ten complete datasets was further analyzed by nonparametric bootstrapping using 10,000 replications per dataset (Reference Briggs, Wonderling and Mooney3). The 2.5th percentile and the 97.5th percentile of these bootstrap replications form the 95 percent confidence interval of the difference in costs and QALYs. The uncertainty around the point estimates of the incremental cost-effectiveness ratios (ICERs) was displayed by plotting the bootstrap replications in cost-effectiveness planes (CE-planes). In addition, cost-effectiveness acceptability curves were drawn, which show the probability that the INTERCOM program is cost-effective at several values of the willingness-to-pay for one additional QALY (Reference Fenwick, O'Brien and Briggs5;Reference Van Hout, Al, Gordon and Rutten19). All analyses were performed with either SPSS version 13.0 or SAS V8.
RESULTS
Subjects
In total, 175 of the 199 randomized patients were included in this cost-effectiveness study, because they had at least one outcome measurement after start of treatment and completed at least one cost booklet. Mean age was 67 years (SD 7), 26 percent was female, FEV1 percent predicted was 60 percent (SD 16), EQ-5D utility index score at baseline was 0.80 (SD 0.18), and patients had on average 1.5 (SD 1.5) co-morbidities at baseline. Baseline characteristics of patients in the INTERCOM and the usual care group were comparable. Of the eighty-seven patients in the INTERCOM group that were included, all visited the physiotherapist and the respiratory nurses and twenty-one received additional nutritional advice and Respifor®. One hundred fifty-eight patients completed the 2-year study period; 75 percent in the INTERCOM group and 84 percent in the usual care group, which was not a statistically significant difference. Drop-outs in the INTERCOM group had a significantly shorter length of stay in the trial than drop-outs in the usual care group. Besides that, drop-out in the INTERCOM group was related to a more impaired health status compared with completers, which was not the case in the usual care group.
Availability of Data
Information about hospitalizations and outpatient visits to medical specialists obtained from hospital records was available for 171 patients (97.7 percent). All other registrations were 100 percent complete. Eighty-three percent of the 158 patients who completed the study filled in the cost booklet for the exact two year period, while the remaining seventeen percent missed on average 2.6 weeks. The missing number of cost booklets in drop-outs was higher. Seventy-one percent of the seventeen drop-outs did not complete the cost booklets until their formal date of drop-out with an average of 8.3 weeks missing. After the formal date of drop-out the number of weeks with missing data was on average 37.8 per patient. For all data sources, the total percentage of missingness was below 10 percent (Table 1).
Table 1. Mean Resource Use per Patient and Correlations between Self-report and Care Provider Registrations for the Complete 2-Year Study Period before Multiple Imputation of Missing Data (n = 175)
aThe percentage of missingness was calculated as the total number of weeks with missing data summed over all patients divided by the maximum number of observable weeks if all patients had complete data for the entire 2-year study period (=18200).
bOnly applicable to patients in the INTERCOM group,
cOnly applicable to nutritionally depleted patients in the INTERCOM group.
COPD, chronic obstructive pulmonary disease.
Agreement
For all types of resource use, the mean unimputed resource use as obtained from the registrations was higher, except for visits to the physiotherapist, for which the mean number of visits obtained from the cost booklet was slightly higher (Table 1). Agreement was almost perfect for number of COPD-related and total hospital admissions, number of COPD-related and total hospital days and number of visits to the physiotherapists (all rho > 0.8). Agreement was substantial for visits to the medical specialists, the respiratory nurse, and the dietician and the number of units Respifor® used (rho > 0.6), while agreement for COPD-related and total daycare treatment was moderate (rho > 0.4). The percentage of perfect agreement decreased as the mean resource use increased. Agreement did not worsen or improve over time (Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc).
Variables Related to Differences in Cost Based on Self-report or Care Provider Registrations
Comparison of the total unimputed costs between the two data sources showed that 106 of 175 patients (61 percent) were underreporting, that is, they had higher costs based on the registrations compared with the cost booklet. Sixty-five patients (37 percent) were overreporting, because they had higher costs based on the cost booklets compared with the registrations. For the remaining four patients, the absolute difference between the two data sources could not be calculated, because data for visits to the medical specialist and hospitalizations were not available from the registrations. In the multivariate linear regression, the degree of underreporting was significantly independently associated with drop-out and total costs. Patients who dropped out during the trial and patients with higher total costs had larger differences in costs between the registrations and the cost booklet compared with patients who completed the study and patients with lower total costs, respectively. The degree of overreporting was only associated with total costs with higher total costs resulting in more overreporting. The association of drop-out with underreporting was confirmed using the logarithm of costs as the dependent variable. No association was found with treatment group, gender, age, number of co-morbidities at baseline, health status, or indicators of disease severity.
Supplementary Figure 1, which can be viewed online at www.journals.cambridge.org/thc, shows the difference in costs between registrations and the cost booklet after multiple imputation. These are the final cost estimates used in the cost-utility calculations. The cost difference was highest for hospitalizations, approximately 390 euros and lowest for visits to the dietician, approximately 50 euros.
Table 2 shows the mean costs by treatment group after applying multiple imputation separately for costs based on the registrations or the cost booklet. Costs for visits to the physiotherapist, respiratory nurse, dietician and the use of diet nutrition, all elements of the INTERCOM program, obviously were significantly higher for the INTERCOM group, irrespective of the data source used. Costs for visits to the medical specialist were significantly higher in the usual care group based on the registrations, while this was not the case when costs were based on the cost booklet. However, differences between the two data sources were small across all types of resource use (Table 2).
Table 2. Mean Costs per Patient for Different Categories of Care Based on Care Provider Registrations or the Self-reported Cost Booklet after Multiple Imputation of Missing Data (in 2007 euros)
Note. Gray cells contain data obtained from the two different data sources. Data in white cells are based on the same data source either the self-reported cost booklet or care provider registrations. Small differences in the white cells are the result of the multiple imputation procedure.
a Other direct medical costs included costs of visits to other therapists, alternative therapists, social workers and psychologists, home care, ambulance transportation, pulmonary rehabilitation (daycare treatment and inpatient), psychiatric hospital admissions, and medical devices.
b Costs paid by the patient included costs of over-the-counter medication, paid and unpaid household help, and travel expenses.
The difference in total costs between the two treatment groups was comparable for both data sources 2,498 euros (95 percent confidence interval [CI], −88 to 6,084) based on the registrations and 2,444 euros (95 percent CI, −819 to 5,950) based on the cost booklet. The gain in QALYs due to the INTERCOM program was 0.08 (95 percent CI, −0.01 to 0.18). This resulted in ICERs of 29,390 euro/QALY based on the registrations and 29,100 euro/QALY based on the cost booklet. CE-planes for both data sources were similar (Figure 1). For both the registrations and the cost booklet, approximately 88 percent of the bootstrap replications fell in the upper-right quadrant indicating that the INTERCOM program has a higher gain in QALYs, but also higher costs. The acceptability curves were also comparable. The probability that the INTERCOM program is cost-effective at a willingness-to-pay of 20,000 and 50,000 euros per QALY gained was in both data sources 37 percent and 69 percent, respectively.
Figure 1. Cost-effectiveness planes, cost per quality-adjusted life-year (QALY).
DISCUSSION
This study showed the impact of self-report or registration-based resource use on costs and cost-utility. Agreement between self-reported resource use and resource use based on registrations was good or substantial for most types of care. Because inaccuracy increases with longer recall periods (Reference Bhandari and Wagner1), the relatively short recall period in our study may have contributed to this high agreement. The cost booklets were designed to record resource use per week and each booklet covered 4 weeks. The booklets were collected every 2 months. This is a relatively short recall period compared with other studies using recall periods of 6 or even 12 months (Reference Lubeck and Hubert8;Reference Petrou, Murray, Cooper and Davidson12;Reference Roberts, Bergstralh, Schmidt and Jacobsen14;Reference Ungar and Coyte17). The high agreement between the two data sources for hospital admissions/days were in accordance with other studies showing a high agreement for major events (Reference Bhandari and Wagner1;Reference Lubeck and Hubert8;Reference Ungar and Coyte17). The agreement for visits to the physiotherapist was higher than in other studies (Reference Goossens, Rutten-van Molken, Vlaeyen and Van Der Linden6;Reference Reijneveld13), probably because the visits took place on a regular basis, two times a week in the first 4 months and once a month thereafter. Agreement for daycare treatment in hospital was poor. This may have been related to the fact that the cost booklet did not explicitly specify daycare treatment in hospital separately from inpatient hospitalizations including an overnight stay. It identified daycare treatment when the date of admission and discharge was the same.
Our study confirmed that self-report results in underestimation. For all categories of care, except one, mean resource use was lower for the cost booklet than for the registrations. Analyses of the difference in total costs based on either the cost booklet or the registrations showed that both under- and overreporting were associated with total costs. The association between increased visit frequency and underreporting was reported by several studies before (Reference Bhandari and Wagner1;Reference Bellon, Lardelli, Luna and Delgado2;Reference Weissman, Levin and Chasan-Taber20). As total resource use increases patients are more likely to forget visits or unwilling to write everything down. The relation between increased visit frequency and overreporting has also been found in other studies (Reference Petrou, Murray, Cooper and Davidson12). With an increase of resource use, it is more difficult to remember the exact date of a certain visit. As a result visits that occurred outside the actual recall period, may have been included.
The absolute difference in costs between the registrations and the cost booklet was approximately 460 euros. Despite the almost perfect agreement for hospitalizations and hospital days, the cost difference between the registrations and self-reported resource use was highest for this type of care, approximately 390 euros. For visits to the dietician the cost difference was lowest, approximately 50 euros, although agreement for this type of care between the two data sources was only substantial. Hence, good agreement between self-reported resource use and resource use from registrations does not automatically result in good agreement in costs, when unit costs are high. Van den Brink et al. also investigated the effect of different data sources on costs for a limited number of types of care (Reference Van Den Brink, Van Den Hout, Stiggelbout, Velde and Kievit18). They found that cost estimates for medication and stoma care products based on self-report were substantially lower compared with providers’ records. The cost estimates for hospital admissions however did not differ much between the two data sources in contrast to what we found in our study.
The observed difference in total costs of approximately 500 euros between the registrations and the cost booklet within treatment groups did not have an influence on the difference in costs between treatment groups. The cost difference between treatment groups was only slightly different, 2,498 euros based on registrations versus 2,444 euros based on the cost booklet. As a result the CE-ratio, CE-planes and acceptability curves were comparable.
A limitation of our study was that we did not have both data sources for all types of resource use. Although it is common in economic evaluations to combine resource use data obtained from different sources, it is unusual to have multiple sources for a single type of resource use. It is not common practice to validate resource use data obtained from one source with a second source. We collected data from several care provider registrations in addition to the data from the cost booklet for the specific purpose to validate the booklet. Of the two items with the highest costs in our study, that is, medication and hospitalizations, only the latter was available from both self-report and registrations. Information on outpatient medication was only available from the administrative systems of patients’ local pharmacies. Given the length of the study, 2 years, and the large number of different medications used by COPD patients, the choice for registrations was made to limit the burden of data recording by the patients. For other high costs categories, such as “other direct medical costs” and “productivity costs,” getting data from registrations would have been very difficult if not impossible. However, if only items with two data sources would have been included in the cost-effectiveness analysis, the cost difference between treatment groups would have been 730 euros based on registrations versus 704 euros based on the cost booklet, resulting in ICERs of 8,590 and 8,379 euros/QALY, which would not have changed the conclusions.
The final estimate of costs used in the original cost-effectiveness study was based on a combination of both sources. Most resource use information was obtained from the cost booklet except for outpatient medication and oxygen, which were obtained from registrations. For hospitalizations we combined both sources and counted all hospitalizations irrespective of whether they were recorded by patients only, in the registrations only or in both sources. This resulted in higher costs for hospitalizations compared with the data presented in this study and, therefore, in somewhat different estimates for the cost difference between treatment groups and the cost-effectiveness, 2,751 euros (95 percent CI, −632 to 6,372) and 32,425 euros/QALY, respectively (Reference Hoogendoorn, van Wetering, Schols and Rutten-van Molken7).
In conclusion, we showed that self-reported resource use led to different cost estimates than care provider registrations, but it did so in both treatment groups. As a result, estimates of the difference in costs between two treatment groups and estimates of the cost-utility of the INTERCOM program were comparable between the two methods of resource use measurement.
SUPPLEMENTARY MATERIAL
Supplementary Table 1: www.journals.cambridge.org/thc Supplementary Table 2: www.journals.cambridge.org/thc Supplementary Figure 1: www.journals.cambridge.org/thc
CONTACT INFORMATION
Martine Hoogendoorn, MSc (hoogendoorn@bmg.eur.nl), Research Fellow, Institute for Medical Technology Assessment, Erasmus MC/University, 3000 DR Rotterdam, The Netherlands
Carel R. van Wetering, (c.vanwetering@wanadoo.nl), Physiotherapist, Department of Physiotherapy, Máxima Medical Centre, 5500 MB Veldhoven, The Netherlands
Annemie M. Schols, PhD (a.schols@pul.unimaas.nl), Professor, Department of Respiratory Medicine, Maastricht University, Medical Centre, 6202 AZ Maastricht, The Netherlands
Maureen Rutten-van Mölken, PhD (M.Rutten@bmg.eur.nl), Associate Professor, Institute for Medical Technology Assessment, Erasmus MC/University, 3000 DR Rotterdam, The Netherlands
