Evidence on Reporting Bias
Over the last few decades, evidence has been accumulating on the existence of reporting biases in the medical literature which “arise when the dissemination of research findings is influenced by the nature and direction of results” (p. 298 (Reference Sterne, Egger and Moher1)). A recent overview found examples of reporting bias in 50 different interventions and procedures in the fields of pharmacology, surgery, diagnostics, and prevention, thus demonstrating that this phenomenon poses a major problem in the assessment of health care interventions (Reference McGauran, Wieseler and Kreis2). The best known form of reporting bias is publication bias, defined as “the publication or nonpublication of research findings, depending on the nature and direction of the results” (p. 298 (Reference Sterne, Egger and Moher1)). However, even in published studies, subsets of the original outcomes or analyses may be selectively reported (outcome or analysis reporting bias) (Reference Higgins, Altman, Higgins and Green3).
Challenge for HTA Agencies
Reporting bias tends to overestimate benefits and underestimate harms of health care interventions, thus leading to potentially biased conclusions in systematic reviews and health technology assessment (HTA) reports (Reference McGauran, Wieseler and Kreis2;Reference Song, Parekh and Hooper4). This was illustrated by a recent assessment of reboxetine by the German HTA agency IQWiG (Institute for Quality and Efficiency in Health Care): the inclusion of data from ten trials, which had previously been unpublished or incompletely published and were provided by the manufacturer only after considerable public pressure, had a significant impact on the assessment's results. IQWiG's conclusion that reboxetine is “overall an ineffective and potentially harmful antidepressant” (p. 8 (5)) stood in contrast to the findings of several earlier systematic assessments that did not combine all of the existing evidence from clinical trials (Reference Godlee and Loder6), and ultimately led to the exclusion of reboxetine from reimbursement in Germany.
Data Accessible to HTA Agencies
Legislation requires manufacturers to grant regulatory authorities such as the U.S. Food and Drug Administration (FDA) comprehensive access to both published and unpublished data for the approval process (e.g., (7)). However, manufacturers are not generally obliged to provide all of the relevant data to HTA agencies for post-approval evaluations (Reference Wieseler, McGauran and Kaiser8). Health policy decisions, for example, on the reimbursement of drugs, are therefore often based mainly on publicly available information with the potential insufficiencies described above (Reference Wieseler, McGauran and Kaiser8–10).
One way of overcoming the selective or nonpublication of studies is their registration in public databases: in 2004, the International Committee of Medical Journal Editors specified that registration in a public trial registry was a prerequisite for a study's publication (Reference De Angelis, Drazen and Frizelle11). In response to public concerns about reporting biases and to legal proceedings (the paroxetine case (12)), several individual pharmaceutical companies, as well as industry associations such as the Pharmaceutical Research and Manufacturers of America (PhRMA), installed their own registries. In 2007, the FDA Amendments Act made registration and public posting of all applicable clinical trials in the registry ClinicalTrials.gov compulsory for the United States (Reference Wood13). In Europe, the publicly accessible EU Clinical Trials Register was launched only recently (March 2011) and contains information on phase 2–4 adult clinical trials and any pediatric clinical trial carried out in the European Union from 2004 onward (14). However, existing regulations for trial registration have several loopholes, particularly as older drugs, although widely used in clinical practice, are not covered (Reference Wieseler, McGauran and Kaiser8;Reference Turner15).
Other strategies used to identify potentially relevant unpublished studies or data include searching for regulatory authority Web sites as well as conference abstracts, as approximately half of trials reported in conference abstracts are never fully published, and publication is associated with positive results (Reference Lefebvre, Manheimer, Glanville, Higgins and Green16). However, it has been argued that publicly available sources do not necessarily contain all relevant clinical trial results for a given healthcare intervention, and that access to these results is still insufficient (Reference Wood13).
As publication is not a dichotomous event, but rather a continuum (Reference Smith17), the distinction between “published” and “unpublished” is not always clear cut (Reference McGauran, Wieseler and Kreis2). In our study, we use the term “published” to refer to articles published in peer-reviewed journals and the term “unpublished” to refer both to publicly inaccessible data owned by industry or study authors and to “gray literature,” that is, studies not published in peer-reviewed journals but nevertheless publicly available in sources such as trial registries, regulatory authorities or published as conference abstracts.
Aim of Our Study
To the best of our knowledge, no detailed analysis is currently available as to how international HTA agencies address the challenge of unpublished data. On the basis of an analysis of publicly available methods papers and HTA reports, the aim of our study was, therefore, to provide an overview of current policies and practices with regard to searching for, including and reporting of unpublished data. Our primary interest was to explore whether and how HTA agencies carry out requests to industry to include data in their assessments that are not publicly available at all. We also explored to what extent alternative strategies are pursued to identify grey literature from trial registries, regulatory authorities and conference abstracts.
METHODS
Eligible HTA Agencies and Documents
We analyzed two different data sources: “methods papers,” that is, documents describing the HTA organizations’ methods for assessing the effectiveness of interventions, and “HTA reports,” that is, full assessments of a health technology or focused assessments of drugs for reimbursement and pricing decisions. We aimed for a broad sample to capture current practice in the HTA community and therefore identified international HTA agencies from four different sources: (a) members of the International Network of Agencies for Health Technology Assessment (INAHTA) as of December 2010, (b) partners in the European Network for Health Technology Assessment (EUnetHTA) project as of December 2010, (c) nonprofit members in Health Technology Assessment international (HTAi) as of May 2011, and (d) HTA agencies included in comparative analyses published in the International Journal of Technology Assessment in Health Care in the years 2009 and 2010. Eligible agencies were those that produced HTA reports and for which we were able to identify both eligible methods papers and HTA reports in English, German, or French (for inclusion and exclusion criteria see Supplementary Table 1: http://dx.doi.org/10.1017/S026646231300072X).
Data Collection, Extraction, and Analysis
We searched all Web sites of potentially eligible agencies at the beginning of 2011, both to check whether they are directly involved in HTA by either commissioning or conducting assessments and to identify methods papers as well as HTA reports. We contacted agencies between February and May 2011 to verify the information obtained online and confirm the completeness and currentness of the documents retrieved. We did not contact agencies where the Web site clearly indicated that they are not involved in HTA production or where insufficient information in English, French, or German was available.
Methods Papers
We included methods papers developed by the respective agencies that were publicly available and described methods for assessing the effectiveness (i.e., not only cost-effectiveness) of drug or nondrug therapeutic interventions. We included more than one methods paper if they had different scopes, yet only those for which eligible reports were also publicly available. If further documents explicitly addressing the inclusion and handling of unpublished data were available, they were also included in the analysis.
HTA Reports
We included HTA reports that were publicly available in full (i.e., not only in summary) and reported a systematic evaluation (i.e., based on a search for literature in at least two sources) of the effectiveness of drug or nondrug therapeutic interventions from January 2006 onward. We excluded reports that (i) were commissioned by another agency that was also included in our sample, (ii) were based solely on secondary publications (e.g., “reviews of reviews” that did not include primary studies), or (iii) were draft versions. We included the three most recent reports on both drugs and nondrug therapeutic interventions or fewer if less than three were available.
To reduce the heterogeneity between reports and due to limited resources, we did not examine documents (neither methods papers nor HTA reports) reporting the evaluation of diagnostic procedures or population-based preventive interventions. In addition, for agencies producing different types of documents, we restricted our analysis to one type (e.g., to multiple technology assessments for NICE documents) to reduce the complexity of the analysis.
We developed and pilot-tested sheets to extract information from methods papers and HTA reports. Information was extracted by one person (JK) and checked by a second (DP) for the categories depicted in Supplementary Table 2, which can be viewed online at http://dx.doi.org/10.1017/S026646231300072X. Disagreements were solved by discussion and by consulting the original documents again. We provide a qualitative analysis of information obtained from methods papers to show the range of different approaches currently in place at the agencies in our sample and descriptive statistics for the analyses of HTA reports.
RESULTS
Agencies Included in Our Sample
A total of 121 agencies were potentially eligible and eighteen fulfilled all inclusion criteria (see Figure 1). The agencies included (see Table 1) are diverse with regard to their scope and their role within the respective health care systems.
Figure 1. Flow chart of selection of HTA agencies (May 2011).
Table 1. Agencies Included in the Analysis and Approaches Regarding Requests to Industry
Note. Explanation: 1 = Manufacturers are routinely approached; 2 = Manufacturer requests may be carried out at the discretion of the HTA authors; 3 = Manufacturer requests are not explicitly mentioned in the methods paper; 4 = Manufacturers (or others) file an application with their data for the product to be assessed.
The information reflects the content of the methods papers available to us at the time of our search. Note: HAS bases its assessments on a review of the literature and dossiers from pharmaceutical companies. The only methods papers that we were able to identify, however, did not reflect this current practice.
Methods Papers Describing Efforts to Include Unpublished Data
Methods papers published by eligible agencies varied considerably regarding length (ranging from 9 to 294 pages), as well as currentness, with the oldest documents having been published in 2000, and the newest ones published as drafts in 2011 (for a complete list, see Supplementary Table 3, which can be viewed online at http://dx.doi.org/10.1017/S026646231300072X). Some of the agencies contacted informed us that the methods papers were being revised at the time of our research or no longer reflected current practice and more recent, internal guidance was being followed instead. However, in the absence of more up-to-date, publicly available documents, these methods papers were still included. The documents also reflect the scope of the respective agency, thus being comprehensive or addressing exclusively the evaluation of drugs or medical devices. Some of the documents are specific guidelines for the respective agencies (e.g., DERP, LBI [for agencies’ full names, see Table 1]), while others have a broader target audience (e.g., DACEHTA, CRD). Where guidance for obtaining data was given separately for assessing effectiveness and adverse effects (e.g., CRD, AHRQ), we limited our analysis to the former.
Requests to Industry
We found that HTA agencies pursue different approaches concerning the question as to whether data are requested from industry in the course of HTA production. These can be grouped into four broad categories: (i) manufacturers are routinely approached as part of the search for relevant literature (seven agencies); (ii) manufacturer requests may be carried out at the discretion of the HTA authors (three agencies); (iii) manufacturer requests are not explicitly mentioned in the methods paper (six agencies); (iv) manufacturers (or others) file an application with their data for the product to be assessed (two agencies). Differences between agencies’ approaches to a considerable extent also reflect the respective national legal context (e.g., when assessments are based on submissions by manufacturers instead of comprehensive searches carried out by HTA agencies) (see Table 1).
Five of the seven agencies that routinely make requests to manufacturers provide details of these processes in their methods papers; these are summarized in Table 2.
Table 2. Standardized Formats for Requests to Industry
Note. The information reflects the content of the methods papers that were available to us at the time of our search (see Supplementary Table 3 for a full list of references).
Conditions for accepting and including unpublished data in reports vary between agencies. The following conditions were addressed by at least on agency: completeness, adequate detail, non-confidentiality, and the timely submission of the data (see Table 3 for examples).
Table 3 Conditions for the Acceptance of Unpublished Data (Examples)
Note. The information reflects the content of the methods papers available to us at the time of our search (see Supplementary Table 3 for a full list of references).
Few documents contain detailed guidance on how to report requests to industry: CRD provides an example for the documentation of a request, CADTH foresees requests to manufacturers as a standard part of the documentation of the literature search, DERP explicitly states that companies providing dossiers should be accounted for in the results section of the report, and IQWiG also states that requests to industry have to be documented in the reports. NICE publishes a list of consultees on its Web site. Several agencies (e.g., GÖG, KCE) state that the literature search has to be documented with enough detail to allow reproduction, without explicitly addressing requests to industry.
Equally, only a few methods papers in our sample provide explicit guidance on how to report previously unpublished data in HTA reports: DERP provides complete copies of dossiers to the public upon request once the report is published. IQWiG foresees the possibility to publish short summaries of previously unpublished studies in its reports, as well as any additional relevant unpublished data. The methods papers of both agencies point out that all previously unpublished data included in the analyses are reported. If a submission to NICE contains previously unpublished confidential information, the manufacturer has to provide a second version with the confidential information removed, which is then published on the NICE Web site as a supplementary document together with the assessment. For previously unpublished trials, NICE requests a structured abstract for public disclosure. Information that has been submitted in confidence is removed from the actual HTA report by blackening the relevant passage. Several agencies (e.g., PHARMAC, MSAC, TLV) point out that the respective legal requirements concerning the disclosure of confidential information apply.
Searches in Publicly Available Sources
Trial registries, regulatory authorities and conference abstracts all seem to play a similar role as additional sources: routine or optional searches in these sources are described in the methods papers of 9, 11, and 8 of eighteen agencies, respectively (see Supplementary Table 4, which can be viewed online at http://dx.doi.org/10.1017/S026646231300072X). Although the distinction between routine and optional parts of the literature search is not always clear cut, it seems that searching for conference abstracts is less often defined as a routine element. Table 4 depicts options for searching in these sources that we identified across agencies.
Table 4. Options for Searching in Publicly Available Sources
Note. The information reflects the content of the methods papers available to us at the time of our search (see Supplementary Table 3 for a full list of references).
a: This registry which was set up by PhRMA is no longer available.
HTA Reports Describing Efforts to Include Unpublished Data
We identified seventy-three HTA reports published between 2006 and 2011 (thirty-three on drugs and forty on nondrug interventions [one report assessed both drug and nondrug interventions and was analyzed twice]) in our analysis. Nine of the HTA reports in our sample assessed nondrug interventions that do not involve the application of a medical device (for example, psychosocial interventions for people bereaved by suicide); we thus excluded them from further analyses regarding requests to industry and searches of regulatory authority Web sites.
Requests to Industry
Current practices described in HTA reports broadly reflect the methods outlined in their methods papers: of those agencies with routine requests (see above), twenty-one of thirty-one eligible reports (68 percent) explicitly mentioned that requests to industry were carried out. While it remained unclear in three reports whether any response was received, eighteen reports described the receipt of information, although in some cases not all of the manufacturers contacted responded. Half of these responses (n = Reference Laurance9) apparently yielded data previously not publicly accessible; one explicitly stated that no additional information had been retrieved, and for eight reports it was not clearly described.
Of those agencies where requests to industry are at the discretion of the HTA authors, only four of nine reports (44 percent) described a corresponding request; in all cases it remains unclear whether a response was received. None of the seventeen reports of agencies where requests to industry were not mentioned in the methods paper described a corresponding request.
An analysis of the extent to which previously unpublished data were included in the statistical analyses and their possible impact on results turned out to be unfeasible in view of this sparse reporting.
Seven reports were included of those agencies where manufacturers (or others) file an application for the product to be assessed (HAS, MSAC, PHARMAC). While three reports of MSAC did not include unpublished data (possibly reflecting the fact that these reports are carried out separately from the assessment of the manufacturer's application), two of the three HAS reports included unpublished data. Data submitted to PHARMAC included slides presented at conferences and further data submitted in confidence.
Searches in Publicly Available Sources
In thirty-eight of seventy-three reports (52 percent) in our sample, trial registries were searched. Most reports stated that this was done to identify ongoing trials, but some also described that this served to identify both ongoing and unpublished trials. The registry most often searched was ClinicalTrials.gov. In twenty-five of sixty-four reports (39 percent) involving the use of drugs or medical devices, regulatory authority Web sites (mostly the FDA) were searched to identify further documents. In twelve of seventy-three reports (16 percent), the literature search explicitly encompassed hand searching of selected conference proceedings. Of the remaining reports, many reported searches in specialized electronic databases, for example, Conference Papers Index, Inside Conferences or ISI Proceedings.
DISCUSSION
Policies and Practices Regarding Requests to Industry
Although the neglect of unpublished data can seriously threaten the validity of HTA reports, international HTA agencies currently differ considerably in their efforts to address this issue, both with regard to the question as to whether and how industry is asked for unpublished data and to policies concerning the reporting and publishing of data received. Current practices vary according to the different policies implemented by the agencies, where only a minority of agencies foresee routine data requests to industry. Surprisingly, in those reports where manufacturers were approached, the reporting of the request and of the data retrieved was often sparse. It, therefore, often remained unclear whether and which manufacturers were approached, whether they provided data, and whether or where these were included in the statistical analysis.
Alternative Strategies to Identify Unpublished Data
In approximately half of the reports, the search in bibliographic databases was supplemented by a search in trial registries, although it was often not clearly stated whether the main intent of the search was to identify ongoing studies or studies not (yet) published. Some HTA authors expended considerable effort in identifying unpublished randomized controlled trials from trial registries (e.g., (Reference Yank, Tuohy and Logan18;Reference Fayter, Corbett, Heirs, Fox and Eastwood19)).
The handling of conference abstracts varied across agencies. Abstract publications pose a serious problem for HTA authors, as their inclusion may be important to reduce the influence of reporting bias, yet they may lack important information (Reference Hopewell, McDonald, Clarke and Egger20). While many reports excluded abstracts, some HTA authors decided to include studies on the basis of abstracts alone to ensure that as much evidence as possible was included, as long as enough information for appraising the study's quality was available (e.g., (Reference Fayter, Corbett, Heirs, Fox and Eastwood19;Reference Smith, Carson and Fu21)). It is noteworthy in this context that the methods papers of the HTA agencies often did not specify how to proceed with conference abstracts once identified.
Although this was not the focus of our research, several reports in our sample explicitly discussed the problem of publication bias or missing data (e.g., (Reference Wessling and Ramsberg22–Reference Coleman, Baker and Kluger25)), and some applied strategies such as funnel plots or Egger's weighted regression statistics to investigate possible publication bias (e.g., (Reference Coleman, Baker and Kluger25;Reference Neyt, Van den Bruel, Gailly, Thiry and Devriese26)). While many reports put considerable effort into trying to identify unpublished data by means of different strategies, some HTA authors decided as a matter of principle not to search for unpublished data to maintain the reproducibility of the data included (e.g., (Reference Gorenoi, Schönermark and Hagen27)), or because unpublished data are difficult to search for systematically and exhaustively (e.g., (28)).
POLICY IMPLICATIONS
As noted above, we did not assess the impact of requests to industry for unpublished data on the conclusions of the HTA reports as the current extent and quality of information did not allow for such an analysis. The actual impact of requests to industry therefore remains unclear for the reports in our sample. However, as long as legislation does not require all trials to be registered and the reporting of results in registries is incomplete (Reference Prayle, Hurley and Smyth29), there are strong arguments for such requests.
For those agencies or research groups that consider carrying out requests to industry in the future, some lessons may be learned from those agencies that already routinely do so. As some methods papers highlight, it is important to ensure that all relevant unpublished studies are disclosed upon request (e.g., by requesting corresponding declarations from manufacturers): if data were provided selectively, this would possibly even increase reporting bias. Standardized request forms following CONSORT guidelines (Reference Moher, Hopewell and Schulz30) are a way to facilitate requests to prevent selective reporting; ideally, full study reports should be made available upon request. Efforts should also be made to make data provided by manufacturers publicly available as soon as possible to ensure that assessments are transparent and the ensuing decisions open to scrutiny (Reference Drummond, Schwartz and Jonsson31).
Given the sparse reporting of requests to industry found in many of the reports assessed, it would be desirable to develop reporting standards for such requests and for the information they yield. Although reporting guidelines for systematic reviews require all data sources to be stated (Reference Moher, Liberati, Tetzlaff, Altman and Group32), detailed guidance for reporting requests to industry is currently lacking. We suggest that these should include the manufacturers approached, whether they provided data and the publication status of these data (see also (Reference Niederstadt and Droste33)).
However, some reports in our sample (e.g., (Reference Metcalfe, Burgess and Laking23;34)), as well as further examples such as the reboxetine case (Reference Eyding, Lelgemann and Grouven5) or the Cochrane Review on Tamiflu (Reference Jefferson, Jones and Doshi35), show that routine inquiries alone are often insufficient, and despite considerable persistence and investment of resources, all relevant data may not always be provided, as in the case of Tamiflu (Reference Godlee and Loder6;Reference Payne36).
Ideally, requests to industry will be made superfluous in the future by legislative measures that guarantee HTA agencies access to relevant data, as already implemented for early benefit assessments of drugs in Germany (37). A further groundbreaking initiative in the area of drug assessments is EMA's new policy of releasing marketing authorization documents on request (e.g., clinical study reports), after the decision-making process for the application of the drug in question is finalized (38). This policy has been legally challenged: in spring 2013, the General Court, as a result of complaints filed by two pharmaceutical companies, issued a temporary injunction ordering EMA not to provide any documents until a final ruling is given (39). Despite this, EMA is planning proactive publication of such information depending on the type of data (40). For nondrug interventions, however, the road to full availability of relevant data seems to be much longer. Initiatives to promote the prospective public registration of studies and disclosure of data (e.g., (Reference Godlee41)) may, therefore, remain more important for studies evaluating medical devices or other nondrug interventions.
Strengths and Weaknesses of Our Study
We aimed to create a comprehensive and systematic approach for identifying HTA agencies to obtain an unbiased picture of the HTA community. The analysis of both methods papers and recent HTA reports allowed capturing not only the policies specified by the agencies, but also actual practices.
Challenges we encountered were the considerable intertwining of agencies in their work (i.e., mutual commissioning of reports or mutual use of methods papers), the fact that several HTA agencies produce different types of documents, and difficulties in always clearly distinguishing between “real” HTA reports and assessments triggered by manufacturer applications.
Our study has some limitations. Approximately a quarter of nondrug interventions assessed in the HTA reports of our sample did not involve the application of a medical device so that manufacturers could not be contacted. We did not systematically evaluate whether in these cases other possible owners of data, e.g., research groups at universities, were contacted. Selective publication of studies, however, may be equally (or even more) prevalent in academic research, especially with regard to study designs other than clinical trials. A challenge in the evaluation of interventions not commercially owned is that it is less clear who may have carried out studies or who may be in possession of the data, and the diversity of relevant parties in this field may make requests difficult.
In addition, our analysis, which was exploratory in nature, was restricted to information available in publicly available documents, and, despite our efforts, we may have failed to identify all relevant documents. Some methods papers may also not reflect current policies, as indicated by some of the respondents to our inquiries. In the absence of other information available, we treated methods papers equally, regardless of considerable differences with regard to their comprehensiveness and their currentness. As the consideration of reporting bias in HTA has changed over time, differences between methods papers in this regard may also be related to the time at which they were written. Inevitably, our study does not reflect current activities or discussions at the HTA agencies that are yet unconsidered in official documents. Thus, future research could strive for a more extensive participatory approach to expand our knowledge on strategies identified in our study and agencies’ experiences.
CONCLUSIONS
It seems self-evident that HTA reports influencing health care decisions for millions of patients should be based on all relevant data. To different extents, HTA agencies search trial registries, regulatory authorities’ public documents or conference abstracts to identify studies and data not (yet) published in peer-reviewed journals. In future, HTA agencies should be guaranteed comprehensive access to relevant data either by compulsory registration of study protocols and comprehensive results in public trial registries or by being granted access to clinical study documents submitted to regulatory authorities. Until then, requests to industry may represent one strategy to access and include unpublished data, while agencies can learn from each other concerning successful practice. Furthermore, the development of reporting standards for such requests and for data obtained from manufacturers would help to improve the transparency of HTA reports.
SUPPLEMENTARY MATERIAL
Supplementary Table 1–4: http://dx.doi.org/10.1017/S026646231300072X
CONTACT INFORMATION
Julia Kreis, Dipl.-Psych., (julia.kreis@iqwig.de), Research Associate, Institute for Quality and Efficiency in Health Care (IQWiG), Cologne, Germany
Dimitra Panteli, MD, MscPH, Research Fellow, Department of Health Care Management, University of Technology, Berlin, Germany
Reinhard Busse, MD, MPH, FFPH, Professor and Department Head, Department of Health Care Management, University of Technology, Berlin, Germany
CONFLICTS OF INTEREST
Reinhard Busse has received travel support from the European Observatory on Health Systems and Policies. Julia Kreis is an employee of IQWiG. In order to produce unbiased health technology assessment reports, the institute depends on access to all of the relevant data on the topic under investigation. Kreis therefore supports the mandatory worldwide establishment of trial registries and public access to clinical study reports. The other authors report they have no potential conflicts of interest.
