Large clinical trials are the standard for making treatment decisions, and nonpublication of the results can lead to bias in the literature, contributing to inappropriate medical decisions (Reference O'Shaughnessy26). Increasingly, oncology trials are stopped early (Reference Trotta, Apolone and Garattini44) with rates having more than doubled since 1990 (Reference Montori, Devereaux and Adhikari24). In the past 3 years, over 78 percent of randomized controlled trials (RCTs) have used an interim analysis for registration purposes (Reference Trotta, Apolone and Garattini44).
Positive results appear to be published quicker than null or negative results, and only 63 percent of clinical trials are published in full (Reference Scherer and Langenberg33). Nearly one-third are not fully published 5 years after being presented as abstracts (Reference O'Shaughnessy26). Abstracts can only present limited information (Reference Wakelee47) and inconsistencies between conference abstracts and later published reports have been identified, impacting on final assessment results (Reference McIntyre, Moral and Bozzo20).
In recent years, there have been increasing numbers of specialized anticancer treatments. In the United Kingdom (UK), the National Institute for Health and Clinical Excellence (NICE) has issued guidance on the use of such treatments to the NHS. This occurs through NICE's Single Technology Appraisal (STA) program, which aims to assess effectiveness early to allow release of guidance around the issuing of marketing authorization. Chemotherapy drugs have been among the first to be appraised through STAs. However, early appraisal often involves a limited evidence base, with few trials undertaken or published in full. On occasions, details of RCTs may never be fully published.
NICE has issued guidance on several drugs for breast cancer (http://www.nice.org.uk/). Many more targeted therapies, currently in the preclinical testing stage, are likely to emerge for use as combined therapies with existing cytotoxic drugs for breast cancer (Reference Carmichael6). Although these treatments may be beneficial, they could also increase the costs to the health service (Reference Schrag34) and as such timely appraisal is required. Inevitably, such appraisals rely on the availability of good quality evidence on the benefits, harms, and costs of the intervention to allow independent assessment.
The main objective of this systematic review was to identify the delay between conference abstracts and full publication of results from RCTs of new anticancer agents for breast cancer. The secondary objective was to identify whether there are any apparent biases in the publication and reporting of these trials.
METHODS
We identified eleven separate pieces of guidance from NICE for eight anticancer drugs. These were then limited to drugs that had been, or were due to be, appraised under the NICE STA program. Six interventions fitted the criteria (Table 1). Six databases were searched including Ovid MEDLINE and the Cochrane library. Ongoing trial databases were searched for trials in progress. Bibliographies of retrieved articles were also checked (search strategy available on request). Searches for those interventions with previous technology assessment for NICE were limited to studies published after the cut-off dates of searches in these previous publications, until August 2007.
Table 1. Inclusion Criteria for the Systematic Review
aAlone or in combination according to licensed indications
bLapatinib and bevacizumab were ‘appraisals in progress’ therefore indications considered here reflect those identified in the literature for bevacizumab, and the combination in NICE's scope for lapatinib.
RCTs published for the six anticancer drugs for adults with breast cancer were sought. No restrictions were placed on the outcome measures or comparators used at this stage. RCTs were quality assessed using recognized criteria (7). Inclusion criteria (Table 1), decisions on quality criteria and data extraction were applied independently by two reviewers, with any differences in opinion resolved through discussion. For full details, please see Takeda et al. (Reference Takeda, Loveman and Harris42).
RESULTS
Interventions Included
Of the six breast cancer treatments, docetaxel, paclitaxel, trastuzumab, and gemcitabine have been appraised by NICE, whereas lapatinib and bevacizumab were, at the time of writing, appraisals in process. We therefore reported all of the treatment combinations identified in the literature for bevacizumab, and restricted lapatinib to the treatment combination described in the ongoing STA's scope. For docetaxel and paclitaxel, an additional criterion required the diagnosis to include node-positive disease (as per the NICE guidance).
Searches generated 1,556 references (Figure 1). Of these, seventy-one publications were retrieved and screened for inclusion. Only forty-one publications from eighteen RCTs met the inclusion criteria. The eighteen RCTs consisted of two RCTs each for paclitaxel and gemcitabine, three each for docetaxel, trastuzumab, and lapatinib, and five for bevacizumab.
Figure 1. QUOROM flow chart of the systematic review process.
Assessment of Mean Time Between Publication of Abstracts and Full Paper
For this assessment, only those publications reporting the same outcomes were included. Calculation of time to publication was restricted to measures of overall survival (OS) or aspects of disease progression. Some trials reported outcomes in multiple abstracts and full publications. Where this has occurred, careful matching of each abstract with its respective full publication was made and a calculation undertaken. Abstracts which only reported baseline characteristics, adverse events, or quality of life scores were not included in the analysis. The mean time delay to full publication of the RCTs was 9 months (range, 5 to 19 months; Table 2).
Table 2. Abstracts with and without Full Publications
aas of August 2007.
Some trials reported key outcomes in abstract form only (i.e., no full publication of results identified). For these trials, a calculation of the mean time between publication of the abstract and August 2007 was made.
We identified eleven trials without full publication of data presented in an abstract or conference proceeding (Table 2). The duration between publication of the abstracts and the end of our searches varied from 3 to 38 months. The mean time awaiting publication to the end of August 2007 was 16.5 months; however, this estimate is based on a small sample with a large range. The calculation does not account for any differences in the interventions, manufacturers, trial sponsors, or any publication bias. Seven trials were without full publications at least 12 months after the abstract data were presented, and four of these remain unpublished after 21 months or more.
For those studies reporting a full publication, this occurred between 5 and 19 months after the abstract; however, the majority of studies had still not published data in a full publication after at least 12 months.
Comparison of Results of Abstracts and Full Papers
Of the eighteen RCTs, only three had a conference abstract and full publication sharing a common outcome (Reference Jackisch, von Minckwitz and Eidtmann14;Reference Piccart-Gebhart, Procter and Leyland-Jones30;Reference Sartor, Peterson and Woolf32;Reference Smith, Procter and Gelber39;Reference Smith40;43;Reference Von Minckwitz, Raab and Schuette45). Of these, two sets of publications from the HERA trial, with two different abstracts (Reference Smith40;43) linked to two full publications (Reference Piccart-Gebhart, Procter and Leyland-Jones30;Reference Smith, Procter and Gelber39), reported data on OS and time to disease progression (TTP). Of the other two linked studies, one was a publication of a secondary outcome (GEPARDUO—pathological complete response) (Reference Jackisch, von Minckwitz and Eidtmann14;Reference Von Minckwitz, Raab and Schuette45) and one a subgroup analysis of radiotherapy delivery (INT 0148) (Reference Sartor, Fitzgerald and Laurie31;Reference Sartor, Peterson and Woolf32). However, this review only considered data extracted from primary outcomes. Trials usually reported interim analyses of their data in an abstract and full analysis in another linked publication. The interim analysis of data in the HERA trial for OS and for TTP was the same in the abstract (43) and the linked full publication (Reference Piccart-Gebhart, Procter and Leyland-Jones30). The 2-year follow-up analysis of data from patients in the HERA trial, was also included in both the abstract (Reference Smith40) and the corresponding full publication (Reference Smith, Procter and Gelber39). In general, very few studies appear to report the same outcomes in both abstract and full publication, but for those that do, results are the same.
Trials Reporting Interim Results in Abstracts and Final Results in Full Publication
It may not be meaningful to compare interim and final results, however, it is meaningful to consider if the direction of the results is similar. Three trials reported interim data in an abstract and final data in a full publication, two trials for paclitaxel (Reference Henderson and Berry12;Reference Henderson, Berry and Demetri13;Reference Mamounas17;Reference Mamounas, Bryant and Lembersky18) and one for docetaxel (Reference Martin, Pienkowski and Mackey19;Reference Nabholtz, Pienkouski and Mackey25). Although the docetaxel trial reported a second interim rather than a full final analysis, it has been included here as it reports the same outcome measures as the abstract. Data presented for OS in the INT0148 trial was positive for treatment with paclitaxel in both the abstract (Reference Henderson and Berry12) and the full results (Reference Henderson, Berry and Demetri13), with a better effect on survival in the interim analysis. Although the abstract stated that the addition of paclitaxel had a significant impact on disease-free survival (DFS), data for TTP was only reported in the full publication. The NSABP-B28 trial reported no statistically significant differences between treatment arms in survival or death at the interim analysis (Reference Mamounas17), while the full publication (Reference Mamounas, Bryant and Lembersky18) reported a nonstatistically significant reduction in the death rate. DFS in this trial was reported as not statistically significantly different in the interim analysis, but statistically significant in favor of paclitaxel at the full analysis.
For docetaxel, the BCIRG 001 trial reported data for OS and TTP as interim data (Reference Nabholtz, Pienkouski and Mackey25) and the second interim analysis in a peer review publication (Reference Martin, Pienkowski and Mackey19). For OS, the risk ratio (adjusted for node status) was not statistically significant in the abstract (Reference Nabholtz, Pienkouski and Mackey25), but had reached statistical significance by the 5-year results reported in the full publication (Reference Martin, Pienkowski and Mackey19). The risk ratios for DFSl (adjusted for node status) presented in both the abstract (Reference Nabholtz, Pienkouski and Mackey25) and full publication (5-year data) (Reference Martin, Pienkowski and Mackey19) were statistically significant.
A mixed picture appears when assessing the direction of effect between interim analyses and subsequent final analyses. In some trials, the direction of effect was the same, while it was not so for others.
Likelihood of Publication in Relation to Outcome
Only six of the eleven trial abstracts reported OS or an outcome measuring TTP. In this small sample of RCTs, the statistical significance of results did not appear to affect the likelihood of full publication of data previously reported in a conference abstract.
DISCUSSION
Overall, very few of the identified trials had both a conference abstract and a full publication which reported the same outcomes. Only three of eighteen RCTs had one or more full papers which reported the same outcome measures and stage of analysis as earlier conference abstracts. Selective reporting of outcomes in abstracts has been put as high as 76 percent (Reference Al-Marzouki, Roberts and Marshall1), creating a biased picture of the efficacy of the treatment. Abstracts or conference presentations are not subject to a detailed and formal peer review, and provide insufficient evidence to allow for rigorous appraisal of their methods (Reference Shields36). They generally do not include all the information needed to assess the trial, are sparse on study characteristics and offer generalized results (Reference Moher22). The absence of full trial details makes it difficult to determine the value of the research and without all the evidence, could lead to erroneous conclusions being drawn. Furthermore, the rising trend of oncology trials being stopped prematurely increases the potential for unreliable findings to be transferred into clinical practice.
Docetaxel, paclitaxel, and trastuzumab all had at least one full publication reporting OS before NICE guidance being issued, although the OS data for the HERA trial appears to have been only an interim analysis. For gemcitabine, no fully published data on OS was publically available before NICE guidance being produced. At the time of the review, NICE had not issued any guidance on the use of bevacizumab or lapatinib.
This review only considered evidence available in the public domain, while the NICE Appraisal Committee has an additional submission from the manufacturer. Such submissions may contain unpublished data of trials only available publicly as conference abstracts. Although the evidence reviewed by NICE extends beyond that in the public domain, there is still the issue of whether or not such data is of the same quality as that published in peer reviewed journals.
There did not appear to be any particular biases in terms of whether positive results were more likely to be fully published than negative ones. While this is based on a small number of studies and consequently lacks statistical analysis, this is in line with findings by Dickersin et al. (Reference Dickersin, Olson and Rennie9).
We found a mean time of 9 months for trials that had fully published results, and for those not yet published in full, a longer mean delay of 16.5 months. An average delay of 2 years between study termination and the publication of reports has been reported (Reference Trotta, Apolone and Garattini44). However, breast cancer trials have been found to have the highest proportion of unpublished studies at 36 percent at 5 years after publication of the abstract, compared with 26 percent for all cancer (Reference Krzyzanowska and Pintilie15). Publication delay or failure to publish current research can lead to a biased pool of evidence, creating problems for systematic reviewers of new health technologies (Reference Krzyzanowska and Pintilie15). As systematic reviews are generally the starting point for the development of evidence-based guidelines, this bias impacts on their validity (Reference Moher22) and subsequent technology appraisals. While it may be important to include abstracts in systematic reviews, they should be treated with greater caution to counter for evidence of inconsistencies between abstracts and full publications (Reference Trotta, Apolone and Garattini44).
The most frequent reason given by authors for not publishing in full are a lack of time, funding or resources (Reference Krzyzanowska and Pintilie15). Publication bias can cause detrimental effects on the scientific progress and there are implications for human health if only half of scientific results are communicated (Reference Shields36). Furthermore, authors have a responsibility to their participants and failure to publish violates that trust and is considered by some as scientific misconduct (Reference Chalmers8).
Many new technologies, including drugs, will have only recently gained regulatory approval and the complete body of evidence may not be available for public scrutiny (Reference Moher22). Rapid and full publication of trial results is not only vital for the effective appraisal of the efficacy of new technologies upon which treatment decisions are made, but also of interest to all of those concerned with new therapies, particularly in the fight against cancer.
This review was limited to drug combinations and patient groups appraised under the NICE STA program and resulted in a small sample size. Due to this, no statistical analysis was performed. Only the most relevant outcomes to the NICE process were data extracted. Strengths of this review are that the authors have no vested interest in these interventions, and that the principles of good practice in systematic reviewing have been followed.
Future research may consider other anticancer drugs and should investigate the effect of publication delay on decision making.
POLICY IMPLICATIONS
The NICE STA program is designed to provide an assessment of therapies around the time of license. It is essential that the correct decision is arrived at after consideration of the full evidence base; albeit there is a risk that nonpublication or publication delays of evidence will produce a biased assessment. Nationwide-recommendations may be based on an incomplete evidence base and here we provide evidence used in six historical appraisals, which show that delays in publication appear to be common.
CONTACT INFORMATION
Petra Harris, MSc (pharris@soton.ac.uk), Research Fellow, Andrea Takeda, PhD (A.L.Takeda@soton.ac.uk), Senior Research Fellow, Debbie Hartwell, PhD (d.hartwell@soton.ac.uk), Research Fellow, Emma Loveman, PhD (Emma.Loveman@soton.ac.uk), Senior Research Fellow, Southampton Health Technology Assessments Centre (SHTAC), University of Southampton, First Floor, Epsilon House, Enterprise Road, Southampton Science Park, Southampton SO16 7NS, UK
