Hostname: page-component-745bb68f8f-mzp66 Total loading time: 0 Render date: 2025-02-11T15:51:28.833Z Has data issue: false hasContentIssue false
  • English
  • Français

Factors influencing the publication of health research

Published online by Cambridge University Press:  01 August 2004

Eugenia Cronin  and
Trevor Sheldon
Eugenia Cronin
Affiliation:
South East London Strategic Health Authority
Trevor Sheldon
Affiliation:
University of York
Rights & Permissions [Opens in a new window]

Abstract

Objectives: Assess the degree to which research project findings were published and explore factors that influenced publication.

Methods: Questionnaire to project leaders. Classification of publications and findings. Chi-squared; univariate and multivariate Cox regression analyses.

Results: Forty percent of projects published in peer-reviewed journal; highly statistically significant relationships between publication in peer-reviewed journals and (1) projects in Responsive/Fellowships streams (p=.045); and (2) projects awarded >£22,713 (p=.02); influence of study findings not statistically significant.

Conclusions: Funders should consider the significant number of studies that did not result in publication and the higher rate of publication in peer-reviewed journals from some programs.

Keywords

Publication biasMeta-analysisPublic policy
Type
RESEARCH REPORTS
Information
International Journal of Technology Assessment in Health Care , Volume 20 , Issue 3 , August 2004 , pp. 351 - 355
Copyright
© 2004 Cambridge University Press

There is ongoing interest in publication bias, because of its potential effects on the cumulative, accessible knowledge base on which health-care decisions are made. This study examines outputs from a National Health Service (NHS) Research and Development (R&D) program initiated in 1993 that supported health research. In 1991, the NHS established the program in response to concerns about the state of basic and clinical research in England and the push for a health-service culture embracing evidence-based medicine, highlighted by an influential parliamentary review. This program is publicly funded; therefore, investigating outputs that may have affected its impact and payback is important from the point of view of accountability. There is evidence that studies with statistically significant or favorable results are more likely to be published than those with statistically nonsignificant or unfavorable results (4). Publication bias, or the differential rate of submission and acceptance of studies for publication according to their results (1), can result in misleading summaries of the research evidence (4). Research funders are very interested in achieving optimum impact from funded projects. Conditions of funding usually include a requirement for dissemination of results as widely as possible, using such avenues as publication in peer-reviewed journals, books, and presentation at conferences. The NHS R&D program represents one of the first attempts by any country to establish a coherent R&D infrastructure to support a knowledge-based health service (3). Little is known about publication patterns from this program. This study examines factors influencing publication in a cohort of studies commissioned by the North Thames Regional Office from 1993 to 1998.

METHODS

The aim of the study is to assess the degree to which findings from R&D projects funded by this NHS program were published and to explore factors that influenced publication. It uses a cohort of funded research projects to assess publication bias, which is recommended as better than indirect methods, for estimating publication bias (4). The current study was conducted in three parts: (i) Questionnaire to project leaders whose projects finished in the period July 1995–December 1998; (ii) Classification of publications and findings; and (iii) Analysis; Data on a cohort of 101 research projects that finished between 1995 and 1998 were collected from project files and questionnaires completed by project leaders. The sample selected was markedly heterogeneous, reflecting the breadth of the various health research funding programs that had been operating when the studies were commissioned. These programs are described in Table 1.

There were eighty-four responses to the questionnaire. Fourteen of these were excluded because, on review, project end dates had been extended beyond the sample period, publication had not been required at the time, or the project had terminated prematurely. The remaining sample was of seventy studies for which there were completed questionnaires with the required outcome data.

Establishing a classification system proved to be a significant challenge, as much of the previous work in this area has used research cohorts with much greater homogeneity. Like the study by Dickersin and colleagues (2), the current study allowed for classification of findings for which statistical tests for significance had not been used. The eventual classification system followed a decision tree, allowing research studies to be grouped first into research or development, whether they were quantitative or qualitative studies, and finally according to the type of findings.

Descriptive statistical analyses were undertaken, and chi-squared analysis was performed to detect associations between publication and various factors. Next, a multivariate analysis of the rate of publication was undertaken using survival analysis, a method used frequently in epidemiological research. This method required recording the length of follow-up (time between completion of study and the questionnaire) and time until main peer-reviewed journal publication. Cox proportional hazards regression was then used to study the relationship between the time to publication and the set of potential influencing factors (1).

This form of regression accounts for situations where the event of interest (in this case publication in a peer reviewed journal) does not occur for all cases; in other words, there are censored data.

The graphs are presented in the format “1 minus survival function,” reflecting that survival analysis has been applied to a situation for which the subjects (in this case research projects) start off being unpublished and gradually become published, in contrast to the more usual application to patient survival, where subjects start off being alive. This form of presentation reflects cumulative gain rather than cumulative loss.

For each of the analyses, performed on SPSS, the variable “months to main publication” was used as the “time” element, and “publication in peer reviewed journal” was used as the “status” or outcome element. Publication in peer-reviewed journal was used rather than “published at all” because the questionnaire respondents had not consistently reported month of publication in their responses. For peer reviewed journal articles, this date could be ascertained from journal databases, but for book chapters, reports, and other publications, this dating was considered more difficult. As a result, it was possible only to explore timing of publication in the case of studies which resulted in peer-reviewed journal publications.

FINDINGS

Complete data were available for seventy projects. The median study duration was 16 months with a median level of funding per project of £44,361. The median length of follow-up after project completion was 35 months, by which time 40 percent had published in a peer-reviewed journal and 70 percent had published in some form. A total of 12 percent of research projects were responsive (researcher-led or training fellowships), and 78 percent were commissioned according to national or locally set R&D priorities. Ninety-three percent of projects were classified as research and the remaining 7 percent as development projects. A total of 47.1 percent included some quantitative analysis, whereas the remaining 52.9 percent did not.

Analyses of the two-by-two tables undertaken to explore possible associations between a range of independent variables and the proportion that resulted in publication are summarized in Table 2. These tests did not find evidence of an association between publication and whether a study (i) showed an effect, (ii) used statistical tests for significance, or (iii) contained a quantitative element. Two interesting associations between characteristics of studies were found. First, there was a positive association between whether a study showed an effect, and whether it used statistical tests of significance (χ2 (1) 6.275, p=.01), indicating that studies which used statistical tests of significance are more likely to have reported an effect in the study (bottom of Table 2). Second, we also found a strong linear correlation between the length of the project (in months) and the amount of funding the project received (r=.6; p=.01). Thus, when developing the multivariate model, we excluded “duration of project” and “use of statistical tests,” including instead “size of funding” and “showing an effect,” respectively.

The variables were explored one by one (univariate Cox regression). Several variables showed a statistically significant association with publication in peer-reviewed journals (Table 3). Backward stepwise Cox regression was used to explore which of the variables were the most important influencers of publication pattern. All the factors explored so far were entered, and the best model selected by progressively removing the variable showing the least evidence of an independent association (Table 3).

Beginning with the eight funding programs, variation was noted between the curves, indicating a difference in publication patterns and suggesting that there is a relationship between funding program and publication outcome.

The output showed that, overall, funding program was marginally significantly related (p=.057) to eventual publication in a peer-reviewed journal; however, there appeared to be little difference between individual funding programs. Therefore, the five Commissioned programs were grouped together so that four categories of program remained: Responsive; All Commissioned programs, including Research Implementation; Commissioned—Organization & Management (O&M); Education and Training (ET)/Fellowships (PROG2).

The survival curve indicated a natural break between responsive and ET/Fellowships; and Commissioned programs including O&M. The final best Cox regression model is shown at Table 4, and in Figures 1 and 2. It indicates that the size of the funding (£TWOCAT, which differentiates projects that received up to £22,713 from those that received over this amount) and type of research program under which the project was funded (PROG4, which differentiates Responsive and ET/Fellowship projects from those in Commissioned programs) are both associated with publication rate. Other variables were not shown to be associated with the rate of publication, but a larger sample is needed to explore this.

“1 minus survival function” for patterns 1–2. PROG4: 1, Responsive and Education and Training/Fellowships; 2, Commissioned programs and Organization & Management

“1 minus survival function” for patterns 1–2. £TWOCAT: 1, Up to £22,713 (minimum £1000); 2, Over £22,713 (maximum $141,124)

COMMENT

Statistically significant independent relationships were found between the rate of publication in peer-reviewed journals and both the type of program (Responsive/Commissioned) and the cost of the research. Whether studies showed an effect or the direction of this effect was not statistically significantly associated with rate of publication, although this could be due to type II error.

This is one of the first studies exploring publication of outputs from the NHS R&D program. Two factors clearly affected the rate of publication over a median follow-up period of 35 months after study completion. Researcher-initiated projects were more likely to be published. This finding may be important, given the gradual replacement of responsive funding streams in the NHS in favor of research commissioned in areas of nationally determined priorities. In addition, better-funded projects had a higher rate of publication; however, it is not clear whether this finding reflects a tendency for larger projects to result in peer-reviewed publications or whether researchers capable of attracting greater funds are more successful at publishing.

Importantly, there was no evidence of publication bias, although the numbers were small. This study should be repeated in other parts of the national and regional NHS R&D program and pooled.

This study was undertaken by EC and supervised by TS, as part of a Masters of Science in Public Health degree at St. George's Hospital Medical School, University of London. The support of NHS Executive London, R&D Directorate is acknowledged.

References

Dickersin K, Chan S, Chalmers TC, Sacks HS, Smith H. 1987 Publication bias and clinical trials. ControlClin Trials. 8: 343– 353.Google Scholar
Dickersin K, Min YL, Meinert CL. 1992 Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. JAMA. 267: 374– 378.Google Scholar
Peckham M. 1993 Research for health. London: Department of Health;
Song F, Eastwood AJ, Gilbody S, Duley L, Sutton AJ. 2000 Publication and related biases. Health Technol Assess. 4: No. 10.Google Scholar
Figure 0

R&D Program Descriptions

Figure 1

Univariate Associations between Study Characteristics

Figure 2

Results of Univariate and Multivariate Survival Analyses

Figure 3

Final Output Statistics

Figure 4

“1 minus survival function” for patterns 1–2. PROG4: 1, Responsive and Education and Training/Fellowships; 2, Commissioned programs and Organization & Management

Figure 5

“1 minus survival function” for patterns 1–2. £TWOCAT: 1, Up to £22,713 (minimum £1000); 2, Over £22,713 (maximum $141,124)