Policy makers are often expected to make coverage decisions based on the “best available” evidence, which can, at times, be inadequate. By having a “yes” or “no” decision as the only options, promising technologies may be rejected or ineffective (or unsafe) ones adopted, depending more on political and other pressures than evidence. This finding can perpetuate the problems of scientific uncertainty, underuse and overuse of services, and failure to resolve uncertainty through further evidence generation (Reference Tunis and Chalkidou6).
Arising out of the uncertainty in decision making in any healthcare sector, decision makers are faced with the dilemma of determining which has the greater risk: making available medical procedures that are ineffective or even harmful (Type I error) or denying access to medical procedures that are beneficial and efficient (Type II error). Owing to the long shadow of thalidomide, there may be an overemphasis by decision makers on the avoidance of a Type I error. Additionally, the growing availability of new technology and the resultant cost blowouts may also have biased decision makers against making a Type I error. The combined effect may result in an unacceptable level of denying access to medical procedures that are beneficial and efficient (Type II errors) (Reference O'Malley4).
The level of evidence supporting the safety and efficacy of medical devices is typically less than that available for pharmaceutical products (Reference Tunis, Stryer and Clancy7). From the perspective of device manufacturers, increasing the amount of clinical evidence required for approval or reimbursement would create a barrier to market entry. Manufacturers argue that the device industry is fundamentally different from the pharmaceutical industry in terms of organization size and access to capital, and that the engineering framework supporting continuous device innovation stands in contrast to the pharmaceutical industry's focus on the development and testing of drugs (Reference Reed, Shea and Schulman5).
Coverage with Evidence Development (CED), or interim funding, has been used in Australia for several years. Since it came into being in April 1998, the Medical Services Advisory Committee (MSAC) has granted interim listing on the Medicare Benefits Schedule (MBS) and thus access to public funding, to a total of fifteen applications (as at October 2008).
CED differs from traditional postmarketing evidence generation in that the objective of the additional evidence generation is to reduce uncertainty around a specific aspect of the evidence base and, thus, help to inform further decisions about ongoing coverage, often at predetermined points in the future (see Funding column in Table 1). The role of the decision maker in determining the nature of the research is also expected to be greater than in traditional postmarketing studies (Reference Hutton, Trueman and Henshall2).
Table 1. Medical Services Advisory Committee Applications Granted Interim Funding
In 2003, interim funding (MBS Item number 11820) was granted to M2A® capsule endoscopy for the evaluation of obscure gastrointestinal bleeding in adult patients, the eleventh application to be recommended for interim funding by MSAC. A condition of this funding was as follows: “Interim funding is being provided to facilitate collection of Australian evidence of the long term safety, effectiveness, and cost-effectiveness of this procedure. Data collection and analysis are being conducted by the Gastroenterological Society of Australia (GESA). Continuation of funding is dependent on the progress of this data collection. Therefore providers of this service are strongly encouraged to take part in the data collection process.”
MSAC stipulated that data should be collection over a period of 3 years to generate sufficient evidence of long-term safety, effectiveness, and cost-effectiveness of capsule endoscopy.
The PillCam® Capsule Endoscopy Register that evolved in response to this requirement was faced with overcoming two obstacles: First, there were the practical difficulties of designing an effective questionnaire, getting the physicians to participate, and creating an efficient system for processing the completed questionnaires. Additionally, the data then needed to be analyzed, statistically validated, and a report written. Second, the analysis needed to answer the three key questions posed in the MSAC Assessment Report. (i) Will the mean yield of PillCam® Capsule Endoscopy observed in the clinical studies and applied to the economic model (59.9 percent) be repeated in practice? (ii) Will a positive yield with PillCam® Capsule Endoscopy prevent all further diagnostic procedures in practice? (iii) Are the ongoing treatment costs of obscure gastrointestinal (GI) bleeding at least $683 per patient per year?”
METHODS
This study examines the case study of the interim funding of PillCam® Capsule Endoscopy and the effectiveness of the PillCam® Data Register in terms of how well the objectives of CED were met. How the data were collected, the methodological difficulties associated with the collection and analysis of the data, and the usefulness of the outcomes of the data are examined.
Also discussed are the lessons learned from this case study and how generally applicable this method is in overcoming the problems faced by policy makers in avoiding Type I and Type II errors and the medical device industry in cost-effective evidence-based medicine (EBM).
RESULTS
The Creation of the Register
Given Imaging, the company responsible for the development and supply of the PillCam® capsule, met with members of GESA to discuss and review how best to comply with the data collection condition of the MBS listing. No funding, guidelines, or infrastructure assistance for the data collection was made available from MSAC, the Australian Government Department of Health and Ageing, or GESA.
Independent professional advice and assistance for the establishment of the PillCam® Data Register was obtained, fully supported, and managed by Given Imaging under the supervision and support of GESA. GESA and Given Imaging met with key clinical experts in Australia who led the design of a practical and effective data collection form, fully approved by GESA.
A comprehensive data collection program was established, managed, and regularly updated. Given Imaging provided a dedicated computer, fax number, and staff to receive and update the data submitted to the register. A customized ACCESS-based data software program was created by a third party.
A copy of the PillCam® Data Register outline and associated documentation were provided to every known physician performing PillCam® Capsule Endoscopy at the time. With every PillCam® SB 10 pack supplied, the user also receives 10 printed data collection forms. These forms were divided into two sections: Section A covering the procedural and patient history information, and Section B covering the follow-up. Once completed, these forms were faxed to a dedicated fax/computer located within the Sydney office of Given Imaging and the data entered into the PillCam® Data Collection Register.
The data were processed into summary reports such as individual Doctor ID reports and cohort reports. Part of the feedback process was the provision of comparative reports to the physicians in order for them to compare their data with the full cohort, thus providing them with a valuable quality management tool. All patients and physicians identification were de-identified throughout the entire register.
The data in the register were independently validated. A logistic regression was carried out, and the data were found to be largely free of bias and systematic distortions. Those discrepancies that were identified statistically were referred to Given Imaging and in all cases were attributable to differences in the patient mixes of the physicians' practices. In some cases these differences were substantial, but when referred to a normed sample with the same presentation characteristics, no significant differences were apparent. In particular, physicians who had reported very few cases did not differ in their findings from those with many more reports, thus enabling the full data set to be considered collectively.
The Register: Clinical and Economic Outcomes
The PillCam® Capsule Endoscopy for the Detection of Obscure Gastrointestinal Bleeding (OGIB) Report, based on data from 2,949 patients collected between 2004 and 2006, was submitted to MSAC in May 2007. The MSAC meeting held in November 2007 concluded that: “MSAC has considered safety, effectiveness, and cost-effectiveness for Capsule Endoscopy for use in obscure gastrointestinal bleeding. MSAC recommended that full public funding be supported under the current MBS Item Number 11820 as capsule endoscopy is as safe as and more effective than comparable diagnostic tests. It is the preferred choice of patients and has the potential to reduce the number and cost of previous investigations.”
In September 2008, the report from the PillCam® Data Register submitted to MSAC was updated to include the final patient population of 4,099 patients submitted by physicians between 2004 and 2007.
The number of patient reports per physician ranged from 1 to 328. The 57 physicians who contributed to the register were classified as being in either a secondary or tertiary referral practice. Just over 88 percent of the patient reports submitted came from a secondary referral practice.
Supplementary Table 1, which can be viewed online at www.journals.cambridge.org/thc, shows the annual number of reports received compared with the total annual number of procedures claimed for capsule endoscopy under MBS Item Number 11820. This table shows that the overall response rate for the 4 years was 28.7 percent with a GI Abnormal detection rate of 70.2 percent. In comparison, the second year of the register with 33.1 percent of the total responses, had a response rate of 44.4 percent and a detection rate of GI Abnormal of 70.9 percent.
Supplementary Graph 1, which can be viewed online at www.journals.cambridge.org/thc, gives the monthly percentage of capsule endoscopy procedures reported to the register. The duration originally set by MSAC was 3 years, approximately May 2004 to May 2007. However, the data collection requirement remained pending the results of the May 2007 report submitted to MSAC, and the final database included data collected up until December 2007. This graph shows that the response rate peaked at 62 percent in early 2005 and fell below the 30 percent level in mid-2006 and a marked decline in participation in 2007.
According to the data collected over the years 2004 to 2007, the mean yield of GI abnormality detection was 70.2 percent, 10.3 percent more than the assumption of 59.9 percent used in the economic model in the MSAC Assessment Report. Supplementary Table 2, which can be viewed online at www.journals.cambridge.org/thc, shows the annual percentage of GI abnormality detection for each of the 4 years of data collection. This table shows that 2005 and 2006, accounting for 33.1 percent and 30.5 percent of the database population, had yields of GI abnormality detection of 70.9 percent and 68.8 percent.
This increased yield of GI abnormalities detected by PillCam® was achieved while maintaining the low rate of complications reported in the MSAC Assessment Report (3). There were complications in 44 of the 4,099 PillCam® Capsule Endoscopy procedures (1.073 percent). The capsule was retained (not passed within a period of more than 2 weeks from date of procedure) in 21 of the procedures (0.512 percent) with 7 (0.171 percent) requiring a procedure to remove the capsule.
As shown in Table 2, before PillCam® Capsule Endoscopy, patients had prior investigational procedures performed in an attempt to identify the cause of the obscure GI bleeding at an average cost of $3,064.
Table 2. Investigations prior to PillCam® and Associated Costs
aNot Medicare Benefits Schedule (MBS). MBS funding for colonoscopy used as an approximate.
Based on the change in the average of before PillCam® investigational procedures from 2004 to 2007, Table 3 shows an average decrease of $967 in the cost per patient of prior investigational procedures.
Table 3. Trend in Total Investigations prior to PillCam® and Average Change in Cost: 2004 to 2007
This decrease is despite the requirement of the MBS listing that: (i) An upper gastrointestinal endoscopy and a colonoscopy have been performed on the patient and have not identified the cause of the bleeding, and (ii) The service is performed within 6 months of the upper gastrointestinal endoscopy and colonoscopy.
The MBS requirement for an upper gastrointestinal endoscopy and colonoscopy to be performed within 6 months before capsule endoscopy was subsequently examined in 2008 by Gilbert et al. (Reference Gilbert, O'Malley and Selby1). They concluded that the yield of repeat endoscopy and colonoscopy immediately before capsule endoscopy is low when these procedures have previously been nondiagnostic and that such an approach is also not cost-effective.
Table 4, based on the follow-up data of the 512 patients with GI Abnormal findings, shows the decrease in the average cost of investigational procedures pre- and post-PillCam® Capsule Endoscopy. The average cost pre was $3,571 compared with $350 post PillCam® Capsule Endoscopy, a decrease of $3,221.
Table 4. Average Cost for Investigational Procedures pre- and post-PillCam® Capsule Endoscopy: GI Abnormal Diagnosis
Supplementary Graph 2, which can be viewed online at www.journals.cambridge.org/thc, shows the growth in the international literature on PillCam® Capsule Endoscopy for the diagnosis of OGIB from 2000 to the end of 2006. There have been in excess of 300 papers published since the MSAC Assessment Report was completed in mid-2003.
CONCLUSIONS
The PillCam® Capsule Endoscopy Register was proposed to reduce the uncertainty surrounding key variables in the MSAC Assessment Report economic model. The three key variables used in the economic model were as follows: (i) the mean yield of PillCam® Capsule Endoscopy, (ii) the change in further diagnostic procedures, and (iii) the change in on-going treatment costs.
The PillCam® Capsule Endoscopy Register provided data that made it possible to validate the assumptions surrounding these key variables. According to the register: (i) The mean yield of PillCam® Capsule Endoscopy in practice was 70.2 percent, significantly higher than the 59.9 percent used in the economic model; (ii) The average cost of diagnostic procedures, for patients with a positive GI Abnormal finding, decreasing from $3,571 before to $350 after PillCam® Capsule Endoscopy; (iii) For the 512 follow-up patients with GI abnormalities found, 40 percent were hospitalized for bleeding in the year before PillCam® Capsule Endoscopy. This rate dropped significantly to just 7 percent in the follow-up period after PillCam® Capsule Endoscopy.
However, perhaps just as importantly, the PillCam® Capsule Endoscopy Register also contributed to answering several of the issues raised in the debate surrounding the use of interim funding and the collection of data and how this “evidence” differs from that generated by clinical trials. It could be argued that the additional evidence for PillCam® Capsule Endoscopy appearing in the literature during the 3-year of interim funding was sufficient to justify MBS funding (Supplementary Graph 2, which can be viewed online at www.journals.cambridge.org/thc). So what were the incremental advantages gained by the data collection?
Unlike pharmaceutical clinical trials, the outcome of a diagnostic clinical trial is influenced by the characteristics of the physician. Concerns were raised that the high diagnostic yield of PillCam® Capsule Endoscopy (59.9 percent) in the literature may have been due to its use by physicians in tertiary referral practices. The register showed a significantly higher diagnostic yield (70.2 percent), despite that just over 88 percent of the patients were diagnosed in nontertiary referral practices. This may indicate that the trialing of PillCam® Capsule Endoscopy in tertiary referral centers, rather than creating a positive bias, was actually a disadvantage.
Supplementary Graph 1, which can be viewed online at www.journals.cambridge.org/thc, shows a decline in participation in 2007, perhaps reflecting one of the main problems associated with a voluntary register continued for an extended number of years. In Australia, the majority of interim funding recommendations by MSAC are for 3 years. However, there does not appear to be any statistical rationale to support the use of this 3-year time limit.
From the physicians point of view, the PillCam® Capsule Endoscopy Register was voluntary and no payments were made for submitting reports. The relatively high response rate, approaching 30 percent, may at least have been partially due to the feedback process of providing comparative reports to the physicians in order for them to compare their data with the full cohort, thus providing them with a valuable quality management tool.
The PillCam® Capsule Endoscopy Register data collection forms were divided into two sections: Section A covering the procedural and patient history information, and Section B covering the follow-up. Of the final 4,099 patients submitted, only 682 had follow-up (16.6 percent). A total of 512 of the 682 follow-ups had GI Abnormal findings (75 percent). This highlights another potential problem and source of bias with the use of data collection for procedures that require follow-up.
The PillCam® Capsule Endoscopy questionnaire was designed by physicians and, as a consequence, may not have been ideal for economic analysis. Despite this, the combination of detailed clinical data and a large sample population made it possible to generate cost savings in the reduction of diagnostic procedures (Table 4). However, although the register provided data on the reduction in hospitalization resulting from the diagnosis by capsule endoscopy and thus treatment, details that would have enhanced the economic analysis, such as length of stay, were not captured by the questionnaire.
The MSAC Assessment Report was based on evidence from only one brand of capsule, PillCam® (previously M2A®), and at the time of the commencement of the register, this was the only capsule available on the Australian market. However, within the first year of interim fundings, a second brand of capsule entered the Australian market, followed by a third brand before the report on the PillCam® Capsule Endoscopy Register was submitted to MSAC in 2007. Despite the availability of the three brands of capsules, only data from PillCam® was collected.
DISCUSSION
The experience of the PillCam® Capsule Endoscopy Register in Australia has demonstrated that data collection can be an effective solution to the unique problems associated with gathering evidence to support medical procedures. This experience has also highlighted several possible improvements in the system.
The financial burden of the register was shared between (i) the Australian Federal Government providing MBS Funding; (ii) Given Imaging, the manufacturer of PillCam® providing the administrative support; (iii) the Gastroenterological Society of Australia (GESA) and key clinical experts designing the data collection forms and overall supervision of the data collection; and (iv) the practicing physicians spending their time completing the forms.
Since its inception in 1998, MSAC has recommended interim funding for fifteen applications with a data collection proviso for most of them. In only two cases was the collection of data funded by the government (endoluminal grafting for abdominal aortic aneurysm and transurethral needle ablation for benign prostatic hyperplasia). In other cases, data have not been collected and have resulted in a renewal of interim funding (brachytherapy for the treatment of prostate cancer) or the provision of permanent funding based on the evidence accumulated from other sources during the time that interim funding was available (deep brain stimulation for symptoms of advanced Parkinson's disease).
Although the PillCam® Capsule Endoscopy Register achieved its purpose, timely up-front guidance may have saved both time and expense. If interim funding coupled with data collection are to become an effective mechanism for bridging the evidence gap, work needs to be carried out by health technology assessment agencies to provide guidance on the design of registers so that they cater for the unique characteristics of individual procedures.
Despite that MSAC has recommended interim funding as a result of fifteen applications and the time limit on many of these has expired, there has not been one example of funding being withdrawn. Further work needs to be done to review all MSAC negative recommendations (procedure not to be funded) and explore the possibility of the expansion of the use of interim funding.
POLICY IMPLICATIONS
Increased use of Coverage with Evidence Development (CED) has the potential to make beneficial new technology available earlier and potentially decrease the probability of Type II errors based on insufficient evidence. The interim funding of PillCam® Capsule Endoscopy is an example of a successful application of CED in Australia. However, the use of CED has at least one major problematic policy implication. What action could and should be taken if the evidence generated does not demonstrate the long-term safety, effectiveness, and cost-effectiveness of the procedure?
SUPPLEMENTARY MATERIALS
Supplementary Tables 1 and 2 (www.journals.cambridge.org/thc) Supplementary Graphs 1 and 2 (www.journals.cambridge.org/thc)
CONTACT INFORMATION
Sue P. O'Malley, BA, MEc, Dip Teach (med.intel@bigpond.com), Consultant Reimbursement Specialist, Medical Intelligence, 13 Cudgee Street, Turramurra, New South Wales 2074, Australia; Doctorate Candidate, Graduate School of Management, Macquarie University, North Ryde, New South Wales 2109, Australia
Warwick S. Selby, MB, BS, MD, FRACP (warwicks@galen.med.usyd.edu.au), Clinical Associate Professor, Faculty of Medicine, The University of Sydney; Senior Visiting Gastroenterologist, AW Morrow Gastroenterology and Liver Centre, Royal Prince Alfred Hospital, Missenden Road, Camperdown, New South Wales 2050, Australia
Ernest Jordan, PhD (ernest.jordan@mq.edu.au), Professor, Director, Higher Degrees Research Marketing & Development, Macquarie International, Macquarie University, North Ryde, New South Wales 2109, Australia
