Conference abstracts generally represent the first reports of novel therapies; thus, they are frequently quoted and widely publicized in the medical and lay media (Reference Hill, Buchbinder and Osborne10;Reference Krzyzanowska, Pintilie and Tannock15). However, some studies show that only approximately half of trials presented as abstracts at scientific meetings are subsequently published in full, meaning that conference proceedings may represent the only published record of randomized controlled trial (RCT) for use by clinicians and in systematic reviews or health technology reports (Reference Herbison9;Reference Hopewell, Clarke and Askie12;Reference Hopewell, McDonald and Clarke14;Reference Scherer, Dickersin and Langenberg18;Reference Scherer, Langenberg and Von Elm19. Trials found in the published literature tend to be larger and show greater effects of a healthcare interventions than those trials found in the gray literature (Reference Hopewell, Clarke and Askie12). There are concerns over the reliability and quality of trials published only in the proceedings of scientific meetings (Reference Hopewell, Clarke and Askie12). Furthermore, the abstracts presented in conference proceedings may be the lasting and only source of information about many trials and their results, which readers will often base their initial assessment of a randomized trial on the information. It is, therefore, important that the report of the conference abstract must have enough details of key elements of quality to enable them to be interpreted properly (Reference Herbison9).
China has modernized scientific research during the past four decades. Research funding and research literature have been increasing yearly since the 1980s. However, little is known by Western researchers about the reporting quality of unpublished conference abstracts of randomized trials in China. This study aims to assess how well or completely randomized trials reported in conference abstracts included in the China National Knowledge Infrastructure (known as CNKI) database in 2007 are reported using the CONSORT for Abstracts checklist criteria (Reference Hopewell, Clarke and Moher13), and whether there is any difference across four key medical specialties.
METHODS
This study is a descriptive study.
Selection of Database and RCTs
We assessed abstracts of randomized trials published in the China National Knowledge Infrastructure database (CNKI) for four key medical specialties, namely internal medicine, surgery, gynecology and obstetrics, and pediatrics. CNKI is one of the largest biomedical databases including national level conference abstracts in China (6). The CNKI database was searched using an extended version of the Cochrane Highly Sensitive Search Strategy (Reference Robinson and Dickersin17) (Table 1) to identify conference abstracts reporting randomized trials. The searching deadline was determined to the end of April 2008. All conference abstracts included in CNKI in the year of 2007 and identified by the Cochrane Highly Sensitive Search Strategy were read to confirm they were indeed reports of randomized trials (i.e., a trial in which the allocation of participants to interventions was described by the words random, randomly, randomized, or randomization). We did not perform hand-searching of the original conference proceedings because in China only participants invited to attend the relevant conferences were able to get copies of conference proceedings. Thus, it is very difficult for us to find printed copies to perform hand-searching.
Table 1. Search Strategy to Identify Reports of Conference Abstracts of Randomized Trials in China National Knowledge Infrastructure database
Assessment of RCTs
We assessed the reporting quality of each conference abstract reporting a randomized trial based on the CONSORT extension for reporting randomized controlled trials in journal and conference abstracts (Reference Hopewell, Clarke and Moher13). (The revised CONSORT for Abstracts checklist is presented in the Table 3.) We assessed each item as “reported” or “not reported” according to whether the author had reported all the contents listed in the refined items or not.
Four assessors (Y.R. Duan, Y.L. Chen, C.L. Ai, and P.X. Chen) independently rated each abstract after discussing and comprehending the items thoroughly with J. Li and M.M. Zhang. We sought further help from S. Hopewell if there was any problem with understanding of the CONSORT for Abstracts checklist criteria. A pilot study on ten papers was performed and inter rater agreement was calculated for each item using the Kappa statistic. Discrepancies were resolved by consensus or discussion. Data analysis was carried out using Excel 2007, Epidata 3.02, and Stata version 9.0. Statistical comparisons of differences between the four medical specialties were assessed using the Yates Corrected Chi-squared test or the Fisher-Freeman-Halton Exact test.
RESULTS
A total of 2055 reports of randomized trials were included in the CNKI database, of which 771 (38 percent) reports were regarding internal medicine, surgery, gynecology and obstetrics, and pediatrics; of these, 567 were identified eligible conference abstracts. Agreement between the reviewers was good with a Kappa score of greater than 0.7. Table 2 presents a distribution of reports of randomized trials across the four medical specialties. Figure 1 presents a flow chart of the eligibility inclusion criteria.
Table 2. The Distribution of Reports of Conference Abstracts of Randomized Trial Reports across Four Medical Specialties in CNKI Database
CNKI, China National Knowledge Infrastructure database; RCTs, randomized controlled trials.
Figure 1. Flow chart of reports of conference abstracts of randomized trial considered for inclusion.
Table 3 presents the CONSORT for Abstract checklist criteria reported in the 567 conference abstracts. Nineteen of the twenty-three items were reported across the included studies, but none were reported across all of the included reports. Only 10 percent (54/567) of included conference abstract reports met a total of eight CONSORT for Abstract criteria. Six conference abstracts reported eleven items and eleven reported only one item. The average rate of reporting was 22 percent.
Table 3. Adapted CONSORT for Abstract Checklist Criteria Reported in the 567 Conference Abstracts of Randomized Trials
Almost all abstracts (94 percent) reported the authors contact details, 83 percent the trial intervention and 78 percent the control intervention, 62 percent reported the participant eligibility criteria and 66 percent the number of participants randomized to each group. Other areas were very poorly reported. Only 1 percent reported the study as randomized in the study title, 2 percent reported the trial design, and only 7 percent reported on blinding. No details of allocation concealment, trial registration, or funding were reported.
Only 36 percent of abstracts reported outcome results in each group, but almost no abstracts stated or described both the estimated effect size of results for each group and the precision of the estimate of effects (i.e., the difference in the effect between the two groups and its statistical significance).
There was no statistically significance difference in the reporting of the primary outcome, the number of participants randomization to each group, the estimated effect size of results for each group, and the precision of the estimate of effects across the four medical specialties. There was a statistical difference for other items but this is most likely due to the small number of trials for these items rather than any clinical meaningful difference across specialties.
DISCUSSION
Searching the CNKI database identified 567 eligible reports of conference abstracts of randomized trials concerning internal medicine, surgery, gynecology and obstetrics, and pediatrics; however, only 1 percent of these could have been identified through searching on the words “randomized” or “trial” in the title of the abstract. Compared with the similar study by Hopewell and colleagues (Reference Hopewell, Clarke and Askie12), the reporting of this item in Chinese abstracts is much lower. Encouragingly, some aspects of trials reported in the CNKI database were well described and the findings of some aspects including the author contact details, details of the number of participants randomized to each arm of the trials, and important information about harms, were higher than or similar to other relevant studies (Reference Abaid, Grimes and Schulz1;Reference Hill, Buchbinder and Osborne10;Reference Hopewell and Clarke11;Reference Krzyzanowska, Pintilie and Tannock15;Reference Xu, Li and Zhang21.
Based on the revised CONSORT for Abstract checklist, our assessment indicates that the reporting quality of conference abstracts of randomized trials has many problems, especially those items concerning the internal validity of a trial, such as the trial design, details of randomization, allocation concealment, blinding, and the flow of participants in the trial. Poor reporting makes it difficult for readers to assess the validity of the trials and correctly use research evidence to guide their clinical decision making (Reference Xu, Li and Zhang21).
Although our study was limited to the conference abstracts from four key medical specialties (internal medicine, surgery, gynecology and obstetrics, and pediatrics) published in China, similar studies in other fields have revealed similar findings suggesting that quality of reporting in abstracts is less than optimal (Reference Bhandari, Devereaux and Guyatt3;Reference Gourgoulianis, Panagakis and Tsakraklides8–Reference Hill, Buchbinder and Osborne10;Reference Hopewell, Clarke and Askie12–Reference Panush, Delafuente and Connelly16;Reference Scherer, Dickersin and Langenberg18–Reference Waxman and Dudley20). A study by Hopewell and Clarke (Reference Hopewell and Clarke11) evaluated RCT abstracts presented at the American Society of Clinical Oncology Conference; no abstract reported allocation concealment, 16 percent reported on blinding, and only 14 percent reported intention-to-treat analysis. A study of all abstracts presented at the 1996 Annual Meeting of the American Academy of Orthopedic Surgeons found that less than half reported key methodological issues, and two-thirds were not followed by a full-text publication (Reference Bhandari, Devereaux and Guyatt3). A recent survey by Berwanger et al. (Reference Berwanger, Ribeiro and Finkelsztejn2) assessed the quality of reporting of 227 trial abstracts of major general medical journals and the findings are suboptimal with only 0.4 percent described allocation concealment; 9 percent clearly specified blinding; 23 percent described intention-to-treat analysis; and 14 percent outlined losses to follow-up. Furthermore, similar studies by Hill et al. and Krzyzanowska et al. (Reference Hill, Buchbinder and Osborne10;Reference Krzyzanowska, Pintilie and Tannock15) analyzed the quality of reporting of RCT abstracts in the proceedings identified with the similar results. These issues of publication bias and inaccuracy have implications for the use of conference abstracts in meta-analyses, where use of this data may influence the outcome of the review (Reference Panush, Delafuente and Connelly16) and make accurate, complete reporting of RCT abstracts at scientific meetings vital (Reference Hill, Buchbinder and Osborne10).
The possible reasons for poor reporting of Chinese conference abstracts of randomized trials may be due to: (i) The CONSORT Statement has only recently been introduced to Chinese medical journals and only four Chinese journals, including Chinese Journal of Evidence-based Medicine, Chinese Medical Journal, Chinese Medicine, and Journal of Chinese Integrative Medicine have recommended CONSORT in their “Instruction for authors or Guide for authors” (http://www.consort-statement.org) (Reference Xu, Li and Zhang21). However, for Chinese authors of conference abstracts and peer reviewers, they know very little about the recent publication of the CONSORT for Abstract checklist. Many studies have proved that clear guidelines such as the CONSORT for Abstracts checklist is needed that outline the key elements of a randomized trial that must be reported in all published abstracts from scientific meetings (Reference Bhandari, Devereaux and Guyatt3;Reference Gourgoulianis, Panagakis and Tsakraklides8;Reference Herbison9;Reference Hopewell, Clarke and Askie12;Reference Panush, Delafuente and Connelly16;Reference Scherer, Langenberg and Von Elm19). (ii) Word limitations placed on abstracts are a frequently cited reason why essential information may be missing from abstracts (Reference Hill, Buchbinder and Osborne10;Reference Krzyzanowska, Pintilie and Tannock15). Authors would provide more details on results and discussion, but cut down on the methodology part (Reference Xu, Li and Zhang21). However, if the trial methodology is not adequately described, the final results are biased. (iii) To overcome some of the issues, inclusion of all RCT on the clinical trials registers was recommended in January 1, 2007, requiring that all prospective, interventional studies must be registered. The Chinese Clinical Trial Register (ChiCTR) was approved by the WHO International Clinical Trial Register Platform in 2007 and many researchers and peer reviewers do not know the importance of prospective trial registration (7), which may be another cause concerning the aspects of poor reporting for trial registration and funding source in this study.
We understand that publication of CONSORT for Abstracts checklist criteria would improve reporting of RCT in abstract form, and, therefore, improve the quality and comprehensiveness of information for abstract reviewers, conference attendees, and researchers. On the other hand, the published abstract is not just a notice about the trial for conference attendees, but will serve at least temporarily as the only published record of the study results. Therefore, adequate reporting of abstracts presented at scientific should be of great concern to the scientific community (Reference Herbison9;Reference Hill, Buchbinder and Osborne10;Reference Hopewell, Clarke and Askie12–Reference Krzyzanowska, Pintilie and Tannock15;Reference Scherer, Dickersin and Langenberg18;Reference Scherer, Langenberg and Von Elm19).
LIMITATIONS
Previous studies have attempted to evaluate the reporting quality of full reports of randomized trials in China using the CONSORT checklist (Reference Bian, Moher and Dagenais4;Reference Chang, Li and Wan5;Reference Xu, Li and Zhang21–Reference Zhang, Zhang and Chang24). However, little is known about the reporting quality of conference abstracts of randomized trials using the CONSORT for Abstract checklist. Therefore, our study has several limitations. First, because we only assessed conference abstracts concerning four medical specialties included in the CNKI database in 2007, these results may not fully represent the report quality of all the Chinese gray literature. Second, our study only focuses on the reporting quality of conference abstracts in 2007; we, therefore, did not find out how trial information reported in conference abstracts differs from their subsequent full publication. Third, as a descriptive study, we did not systematically compare the findings of this study with those of similar published studies.
CONCLUSION
In summary, much needs to be improved in the reporting quality of conference abstracts of randomized trials in China. Since the publication of the CONSORT for Abstract checklist to improve the reporting quality of conference abstracts of randomized trials, we recommend that authors and peer reviewers adopt this checklist to improve the current status of poor quality reporting of conference abstracts, which will enable editors and readers to understand what was done and hence to evaluate the reliability and relevance of the findings.
CONTACT INFORMATION
Yurong Duan, MD (dyr0612@163.com), Chinese Cochrane Centre, West China Hospital, Sichuan University, West China Hospital, Jing Li, MD, PhD (lijing68@hotmail.com), Associate Professor, Department of Evidence-based Medicine and Clinical Epidemiology; Deputy Director, Department of Evidence-based Medicine and Clinical Epidemiology, West China Hospital, Changlin Ai, MD (acl1999@gmail.com), Yaolong Chen, MD (chenyaolong21@163.com), Peixian Chen, (cpxcsugreat@gmail.com), Chinese Cochrane Centre, West China Hospital, Sichuan University, West China Hospital, Mingming Zhang, MSc (mingming-zhang@163.com), Co-corresponding author, Professor, Chinese Cochrane Centre, Sichuan University; Coordinator, Chinese Cochrane Centre, West China Hospital, No. 37, Guo Xue Xiang, Chengdu, Sichuan 610041
Sally Hopewell, PhD (shopewell@cochrane.co.uk), Co-corresponding author, Research Scientist, UK Cochrane Centre, Summertown Pavilion, Middle Way, Oxford OX2 7UK, UK
