Health technology assessments (HTA) are policy-oriented and may have a considerable impact on medical, economic, societal, and ethical consequences of health practices and technologies. This power of HTA reports requires scientists to identify and exclude potential confounders that may affect the validity of HTA reports. Investigating the validity of HTA reports, we realized that different HTA groups use different criteria (3;9) for assessment of the validity of included original papers. This variable may influence the result of the reports and eventually influence the final conclusions. In this study, we compare the methods used for assessment of the validity of the same two original studies included as key studies in two HTA reports and investigate the influence of the assessed validity on the final conclusions of the reports.
METHODS
Two corresponding HTA reports prepared on behalf of the National Institute of Clinical Excellence (NICE) and The Deutsches Institut fuer Medizinische Dokumentation und Information (DIMDI; 6;7) were identified. Both reports were based on the same original papers (1;4). The criteria used by the two groups to assess the validity of the two papers as well as the result of this process were extracted from the HTA reports. In addition, we assessed the validity of the two original studies using the criteria of the Evidence-Based Medicine (EBM) Working Group according to Guyatt et al. and Sackett et al. (2;5). Finally, we investigated the possible effect of the differences in the validity assessments on the final conclusions of the reports.
RESULTS
The list of questions suggested by the EBM Working Group (2;5) to assess the validity of a treatment study (Table 1) was identical to the list of questions used in the NICE report, except two additional questions (presentation of the eligibility criteria and presentation of variance in addition to point estimates for the primary outcome parameters) that were included in the NICE report.
Compared with the EBM list, the DIMDI report (7) asked 18 additional questions shown in Table 2 but did not include questions concerning blinding of patients and physicians, intention-to-treat analysis, and sufficient follow-up observation of the patients.
The catalogue used in the NICE report did not include questions concerning blinding of physicians (only blinding of outcome assessors and patients), sufficient follow-up observation of patients, and the similar treatment of patients, aside from the experimental therapy (Tables 3 and 4).
In addition to the structural variance of the instruments used, an operational variance is recognized, as some of the questions that are identical in all three checklists are answered differently (Tables 3 and 4).
The validity of both key papers demonstrates considerable limitations. In the study by Manns et al. (4), the requirements of correct randomization, concealment of the randomization plan, blinding of physicians, patients and assessors, inclusion of all patients in the analysis, intention-to-treat analysis, the comparability of the groups, and the same treatment of all patients were not or only partially fulfilled. In the paper by Fried et al. (1), the requirements on concealment of the randomization plan; blinding of the physicians, patients, and assessors; inclusion of all patients in the analysis; intention-to-treat analysis; comparability of the groups; and the same treatment of all patients were also not or only partially fulfilled.
In summary, the available data (1;4) cannot support the superiority of therapy with peginterferon and ribavirin over conventional therapy with interferon and ribavirin. There is considerable lack of transparency in both papers (1;4), which could possibly be clarified by examining the original data. Without this clarification, we rank neither of these studies as sufficiently valid to confirm the postulated difference.
In the discussion part of both HTA reports, the authors clearly admit to “limitations” of the validity of the studies included. This important statements, however, are not repeated in the conclusions parts of the reports.
DISCUSSION
In this study, we provide evidence that the authors of HTA reports are using different instruments to assess the validity of the original studies that are included in their HTA reports. We found also evidence that the answers given by different scientists to identical questions—concerning the validity of the studies included—are not necessarily identical, and last but not least found—in both investigated HTA reports—differences in the information provided in the discussion part and in the conclusions.
The last point is most critical, as it may affect the credibility and societal role of HT assessments. It is essential that the scientific community itself sets the quality standards for their products and guarantees the observation of these standards. This statement is true for all types of services of the medical community. More than 10 years ago, an economist from the U.S. general accounting office who attended our conference on “Goals of Palliative Cancer Therapy” predicted:
If physicians will not decide on how to decide, others will…. If the profession does not develop a strategy for resolving the crisis, the courts and legislatures will become increasingly involved in making clinical decisions (8).
At first glance, it seems obvious that an internationally recognized checklist would be helpful in the evaluation of the validity of original papers to achieve uniformity of methodology. The more details of this topic are discussed, the more convinced we are that the available INAHTA checklist contains all important aspects to avoid the three problems described in this communication. The checklist recommends an external review (§4 of the checklist) and emphasizes the important difference that policy-makers, in contrast to scientists, will read only the summary/conclusion of the report (§5). Information that is not expressed in the summary/conclusions will not be recognized by policy-makers. A description of the quality of studies included (§11), the clear presentation of the results (§12), and the interpretation of data (§13), as well as clear conclusions (§13) are also requested in the INAHTA checklist.
Additional recommendations that deal with more details will make the process more complicated but not necessarily better. It is probably sufficient if we, the members of the medical community, guarantee that the available recommendations will be observed and applied. Misunderstood politeness that tolerates incorrectness will support individual but not societal interests. If the self-control of our profession fails to meet the expectations of the society, our profession will loose its autonomy.
CONTACT INFORMATION
Franz Porzsolt, MD, PhD, Head (franz.porzsolt@medizin.uni-ulm.de), Clinical Economics, University Hospital Ulm, Frauensteige 6, 89075 Ulm, Germany
Hana Kajnar, MD, Scientific Assistant, Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany; Medical Doctor, Department of Internal Medicine, Staedtisches Krankenhaus Sindelfingen, Arthur-Gruber-Str. 70, 71065 Sindelfingen, Germany
Aziz Awa, MD, Scientific Assistant (aziz.awa@web.de), Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany
Margrit Fässler, MD, Scientific Assistant (doctores@nikocity.de), Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany; Scientific Assistant, Department of Medical Ethics and History of Medicine, Georg-August University, Humboldtallee 36, 37073 Goettingen, Germany
Barbara Herzberger, MD, Scientific Assistant (bherzberger@aol.com) Human Science Center, Ludwig-Maximilian University, Goethestr. 31, 80366 Munich, Germany
