DISCUSSION

Overall, we found limited evidence of association between methodological features and outcome in the analyses carried out. However, a consistent finding across all the case studies was of no relationship between sample size and outcome frequency. Hitherto, sample size has been used as a criterion for the inclusion or exclusion of case series from reviews. The lack of relationship between study size and outcome suggests that this approach may not be justified. Case series are likely to be more numerous than RCTs or other designs, and there may be an incentive for researchers to limit the number of series included in a review, supported perhaps by the view that this design is necessarily less likely to result in robust conclusions. Our findings tentatively suggest that setting a cut-off in terms of sample size may be less justified than including all studies or taking a random sample.

We found no evidence that prospective series or those in which consecutive cases were enrolled were associated with different outcome frequency to studies not having these features. Again, these criteria are frequently used to judge the quality of case series. These analyses were particularly constrained by inadequate reporting in the original studies. However, all the examples explored were surgical interventions, and it may be that, where retrospective designs were used, case ascertainment was good, reflecting the ease of identifying patients after surgical procedures from hospital records. Where ascertainment is more difficult retrospectively, for example for drug technologies, a greater difference may be shown between retrospective and prospective studies, or those in which recruitment was or was not consecutive. A further consideration in this and all the analyses showing no association between methodological features and outcome is the limited power to detect a significant difference afforded by the small number and heterogeneity of available studies.

In the case of spinal cord stimulation, a significant association between the quality score used by reviewers and outcome was demonstrated. This finding was the only example in which such a score had been used and may suggest that the use of quality scoring systems can differentiate between studies. However, we found no relationship between the individual study factors that made up the score and outcome in the case series. It is difficult, therefore, to conclude whether the score is acting as a valid measure of study quality. Because the relationship between methodological features and validity is not clear and how item scores should be summed into a single measure of study quality remains uncertain, it may be unwise to use such single scoring systems to judge the quality of case series.

Our other findings were inconsistent across the case studies, and the small number of associations demonstrated cannot be taken as good evidence on which to base any change in approach to the appraisal of case series. The finding, in one analysis, that independent measurement of outcome may be important in determining study quality is consistent with the findings of Juni and colleagues regarding blinding in relation to the quality of RCTs (10). This finding is potentially important, but further evidence is required on the importance of this factor in other case series.

The failure to demonstrate any relationship between date of publication and outcome, as with the other negative findings in this study, may be related to limited statistical power. Three other explanations are possible. First, the impact of early adopters and any effects of selection of cases in early studies may be short-lived and, therefore, be not apparent when a longer historical perspective is taken. Second, the effect of the learning curve in the early stages of use of a technology may counteract the effects of case selection. Third, technological improvements may have a very marked effect on successful outcomes.

We suggest that there are complementary positions for different methodological approaches in the ongoing evaluation of health technologies. In some cases, where the natural history of the condition is well understood and a dramatic effect is shown by a technology, comparative studies may not be considered necessary or ethical. We expect that such cases will be very few. It is more likely that case series will continue to be carried out in the early stages of technology diffusion, particularly in surgery where there is a less-stringent regulatory framework governing adoption. Such case series will be important in identifying whether technologies are likely to be efficacious. Early assessment of case series, therefore, may identify technologies that should be subject to more rigorous evaluation. Efficacy may then be established through well-conducted RCTs. This determination, however, may be insufficient to inform practice and policy, and for some technologies, it may be necessary to continue to collect data through case series or, more systematically, through the use of comprehensive registries. These options hold several potential advantages over case series led, more conventionally, by the clinicians delivering the intervention. Standardization of data collection and reporting is more feasible and investigation of the effects of center and operator would be facilitated. Furthermore, establishment of an ongoing system for reporting of process and outcomes would demonstrate changes in the nature of the technology, which is a particular issue in the development of surgical techniques. A key advantage of ongoing collection of data through large case series or registry studies is the identification of uncommon side effects in practice and a high degree of external validity. Using registry or case series data to make a comparison between technologies will continue to be necessarily and severely constrained by the nondirect nature of such comparisons and the effect of a large range of known and unknown confounders. However, the collection of data on the performance of technologies in undifferentiated populations over long time periods will complement and may extend the knowledge yielded in the generally short time scales and selected populations of RCTs.

In the investigation of possible impact of methodological aspects of case series, our examples were all surgical interventions. This means that our findings may not be generalizable to evaluations of other types of technology using case series. As noted above, the effects of learning curve and the possibility of bias arising from enthusiastic early adopters may act in opposite directions, making it difficult to discern any effect relating to timing of publication. A more important problem arising from the nature of the technologies examined is the introduction of further variance in the data as a result of operator effects that would not be apparent in, for example, drug technologies.

The small number of cases examined and the relatively limited number of studies in each set of case series are important limitations to precision and generalizability, which may be addressed by further research. However, it is likely that empirical opportunities for investigation will be few, as has been shown in comparisons of randomized and nonrandomized controlled trials. Under these circumstances, modeling studies may be valuable. Our analysis was necessarily limited to the aggregate reports of individual studies, and there is, therefore, the potential for ecological bias.

Some statistical considerations should be borne in mind when interpreting our results. Weighting for variance is generally favored in meta-analysis and gives greater weight to larger studies to improve precision. Although sample size is not the only determinant of variance in studies, in the current context, it is likely to dominate other factors. As we did not have data on the variance of individual studies, we were constrained to using sample size alone. We found no relationship with study size that does not support concern over increased bias in smaller observational studies. Hence, we have included weighted regression for completeness. Our main statistical approach was robust regression. This technique resists the influence of extreme outliers but results in slightly larger standard errors. Hence, the power of robust regression to detect true differences is slightly reduced compared with ordinary least squares regression. Given the nature of our data, we consider this method to be a reasonable analytic approach. The use of several methods of analysis has led, in some cases, to apparently discrepant results. Given the large number of analyses performed, the usual level of significance of p =.05 should be viewed with caution.

A general problem in the data examined is the very low “signal to noise” ratio. In other words, it is difficult to identify the effects of methodological factors from the effects of heterogeneity between studies in aspects of the populations and interventions. This is a particular problem where reporting of population and intervention characteristics was limited. The impact of unknown confounders, the fundamental reason for favoring RCTs over other study designs, is also an important consideration.

The potential role of publication bias should also be considered. Case series may be particularly prone to publication bias. Being much less robust than comparative or experimental designs, they may be less likely to achieve publication in any journal. Small case series may be more prone to this bias, as in other study designs, and those with less impressive findings and small size are likely to be at greatest risk. However, two findings suggest that publication bias may not be a particular problem in the examples studied. First, the finding of no association between sample size and outcome suggests that smaller studies are not more likely to be positive. Second, the very large range in sample sizes among studies suggests that even small studies are achieving publication. The extent to which these findings are likely to be replicated in other reviews of case series is unknown and further research into the extent and impact of publication bias in different study designs is required.

The general finding of poor reporting of methodological features in case series is a cause for concern and will continue to hamper research into case series and the ability of decision-makers to consider the appropriate influence of case series evidence on policy. We chose to constrain our analyses to reported data, that is, where a methodological feature was not reported in a study, this information was excluded from the analysis. The reporting of other study designs, notably RCTs and systematic reviews, has been improved considerably in recent years. While case series rightly occupy a position low in the hierarchy of evidence, their continued use in health technology assessments seems inevitable and strongly suggests the need to improve reporting quality.