Stephan Leucht and colleagues have over recent years been in the vanguard of using meta-analysis to address key issues and controversies in antipsychotic drug treatment. Their recent article brings three of their most recent meta-analyses to bear on one of the most controversial of these questions in recent years: are the second-generation antipsychotic drugs (SGAs) actually no better than their much-maligned first-generation predecessors (FGAs)?
As we all know, the claims we came to believe in the 1990s were that the non-clozapine second-generation antipsychotic drugs (SGAs) were superior to first-generation antipsychotic drugs (FGAs) in terms of positive symptoms, negative symptoms, mood symptoms, adverse effects, adherence rates and neurocognitive effects. The increased acquisition costs, or price, of the new drugs were worth it in terms of the improved quality of life they delivered and, in any case, these costs would soon be recouped through savings on in-patient stays. These claims were readily taken up by a community of clinicians which had been startled by the unprecedented efficacy of clozapine (Kane et al. Reference Kane, Honigfeld, Singer and Meltzer1988) and were thus primed to accept the idea, which seemed revolutionary at the time, that there could be antipsychotic drugs with efficacy and safety profiles which differed from the conventional drugs. The SGAs, or new atypicals, were seen as a class because they all grew out of a new preclinical model, because they shared a reduced risk of extrapyramidal side-effects (EPS), and because they were marketed as such.
The strengths of Leucht and colleagues' (Reference Leucht, Kissling and Davis2009) meta-analyses is that they are constructed to test specific hypotheses about possible sources of error, such as supposed sources of bias, as well as examining the major questions about superiority in efficacy or safety. In one sense, there is a particular irony to this. It was the repeated deployment of meta-analyses which contributed to the body of evidence purporting to show the clear advantages of SGAs in terms of efficacy and safety, compared to their predecessors. However, there were straws in the wind as early as 2000, when Geddes and colleagues' (Reference Geddes, Freemantle, Harrison and Bebbington2000) meta-analysis suggested that a large part of the variance in the difference between SGAs and FGAs was down to the choice of the comparator drug and its dosage: specifically, haloperidol at rather high dosage. The rest is recent history. Starting with Rosenheck et al.'s (Reference Rosenheck, Perlick, Bingham, Liu-Mares, Collins, Warren, Leslie, Allan, Campbell, Caroff, Corwin, Davis, Douyon, Dunn, Evans, Frecska, Grabowski, Graeber, Herz, Kwon, Lawson, Mena, Sheikh, Smelson and Smith-Gamble2003) trial which showed no advantages to olanzapine compared to haloperidol when used in modest dosages with anticholinergic cover, through CATIE (Lieberman et al. Reference Lieberman, Stroup, McEvoy, Swartz, Rosenheck, Perkins, Keefe, Davis, Davis, Lebowitz, Severe and Hsiao2005), then CUtLASS (Jones et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006), then EUFEST (Kahn et al. Reference Kahn, Fleischhacker, Boter, Davidson, Vergouwe, Keet, Gheorghe, Rybakowski, Galderisi, Libiger, Hummer, Dollfus, Lopez-Ibor, Hranov, Gaebel, Peuskens, Lindefors, Riecher-Rossler and Grobbee2008), the received wisdom about the superiority of the SGAs has been robustly challenged.
One of the often-stated weaknesses of meta-analysis is that it is a hostage to fortune as regards the variable quality of the component trials. So, it is good to see that one of the strengths of the paper is that the authors examine systematically possible sources of bias in the individual trials examined. The issue of industry bias is dissected usefully to show that much of the reason why 90% of industry-funded trials find in favour of the company's compound is down to how respective authors choose to interpret (or spin) the data, rather than the data themselves being clearly in favour of the experimental drug. Another possible source of bias examined is in the choice of a poorly tolerated comparator drug. It emerges after all that there is little straight support for the assertion of the Geddes et al. (Reference Geddes, Freemantle, Harrison and Bebbington2000) meta-analysis that the effect size of the efficacy difference between SGAs and FGAs, specifically haloperidol, was due to high dosing of the comparator drug. Nonetheless, it remains true that most SGA–FGA comparisons have selected haloperidol as the comparator and one cannot help but speculate whether companies tended to choose this drug because of its known high rates of EPS. As the authors suggest, these trials may have thrown up different conclusions if a medium, rather than high, potency FGA comparator had been used. Linked to this is another alleged source of type I error, generating findings erroneously in support of the experimental compound, namely the practice of dealing with missing data-points from drop-outs by using the ‘last observation carried forward’ statistical adjustment. Leucht tested whether this was truly a source of error by reanalysing the original datasets from some trials and finding that it was not. Nonetheless, it would be surprising if this crude statistical technique did not generate bias in some studies, particularly where there were major disparities early on in drop-out rates between treatment arms. Trials such as CATIE and CUtLASS moved beyond this potentially misleading approach, using instead imputational techniques to deal with missing data.
Importantly and more generally, Leucht and colleagues point to widespread design limitations in antipsychotic drug trials. The absence of an agreed criterion for clinical response in schizophrenia contrasts with universally adopted Hamilton-score-based criteria for response in trials of major depression, for example. In schizophrenia trials, important clinical endpoints such as response, remission and relapse have no widely agreed operational definition, which will always hamper summary approaches such as meta-analysis.
Some of the conclusions of the authors might be open to debate. For instance, they suggest that a sufficiently dosed double-blind trial of clozapine versus other SGAs is still needed. Is it? CATIE 2 (McEvoy et al. Reference McEvoy, Lieberman, Stroup, Davis, Meltzer, Rosenheck, Swartz, Perkins, Keefe, Davis, Severe and Hsiao2006) and CUtLASS 2 (Lewis et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006), which were open-label comparisons of clozapine versus non-clozapine SGAs, used blinded raters to overcome the potential bias inherent to open trials. The two trials, although designed differently, showed similar outcomes: a statistically and clinically significant advantage to clozapine over other SGAs. Presumably the authors believe that this observed advantage still might plausibly arise from the non-specific therapeutic effect of regular clinical contact through the mandatory blood monitoring.
Where the authors, and other commentators too, get into a bit of a tangle is over the essentially semantic issue of ‘are SGAs a class?’ The authors conclude that SGAs are not a homogenous category and therefore not a class. As they rightly say, individual SGAs differ one from another in many properties, including efficacy, safety and pharmacology. However, the class of SGAs is explicitly defined by their name: second generation. As such, chronologically defined, they are clearly a class – they were discovered later than the FGAs – and it is precisely this agnosticism which has led to this name being preferred. The term it replaced, ‘atypical’, implied a class defined by either (it was never definitively agreed) a common preclinical mechanism (atypical pharmacology) or a common clinical profile (atypical safety profile). So, SGAs are evidently a class on semantic grounds, but whether it is useful in practice to classify them so, in contrast to FGAs, depends on why one might wish to do it. Shortly after the Leucht et al. article was written, the UK clinical guidelines body, NICE, published its 400-page 2009 update of schizophrenia treatment guidelines supported by a range of systematic reviews. At no stage in the actual guideline is the distinction between FGAs and SGAs employed, and even for first-line treatment there is no explicit recommendation that SGAs are preferred. In considering relapse prevention for example, the guideline states:
All the antipsychotics identified for review have established supremacy over placebo in preventing relapse, although the evidence that any individual drug or group of drugs (FGA v. SGA) has greater efficacy or tolerability is still very uncertain … Any small advantage (offered by SGAs) of reduced EPS may be offset by other adverse consequences not shown by earlier drugs.
The NICE guideline states at the outset that ‘Issues of … affordability are determined by the NHS’. The main distinction at the policy level between FGAs and SGAs has been their acquisition cost. FGAs are cheap, and SGAs less so by at least an order of magnitude. Of course, there are caveats here. The overall costs of health and social care for someone with schizophrenia is high, and drug costs, even for the relatively expensive SGAs, turn out to be a small proportion of this. Moreover, SGAs are one by one becoming generic and, although it will take several years for the price of such generics to approach that of the old drugs, this will blur the usefulness of a distinction on cost. Nonetheless, Leucht and colleagues downplay the issues of cost effectiveness. It is important to realize that both CATIE (Rosenheck et al. Reference Rosenheck, Leslie, Sindelar, Miller, Lin, Stroup, McEvoy, Davis, Keefe, Swartz, Perkins, Hsiao and Lieberman2006) and CUtLASS (Davies et al. Reference Davies, Lewis, Jones, Barnes, Gaughran, Hayhurst, Markwick and Lloyd2007) involved sophisticated cost-effectiveness analyses because the rationale for both studies included issues of relative cost. The initial letters of the CUtLASS acronym stand for ‘cost utility’. Analyses of both studies found FGAs to be the ‘dominant choice’: that is, no less effective, but cheaper. In their conclusion to this section, the authors state that ‘Some may suggest that the money spent on the SGAs should rather be spent on psychosocial therapies. But, unfortunately, the money might be allocated to other areas of medicine such as cardiology, or cancer.’ This could be true, but surely it would be seen as a miserable failure of academic leadership if we were not able to ensure at a national and local policy level that cost savings made from switching to cheaper drugs with equal effectiveness were not reinvested in other evidence-based treatments for patients with schizophrenia.
Looking back over the past 4 or 5 years at the reaction of the field to the data-driven reappraisal of the relative place of SGAs, it is perhaps not too whimsical to see that it has followed the stereotypic phases of a bereavement reaction, and those at the sharp end of the disclosure and discussion of the results of CATIE and CUtLASS can attest to this. First, there was a denial phase: the results were dismissed as due to poor trial design and type II statistical error. Next, there followed a phase of anger. Why were these researchers promulgating such damaging nonsense, which threatened to set the field back 20 years? Most difficult was when this reaction came not from companies, who were usually restrained, nor clinicians, who often were not, but from patient groups. Shortly after the publication and attendant press releases of the CUtLASS 1 trial, the study team took a call from the chief executive of a UK mental health charity. What were we doing, she asked, destroying the hope of better treatment for sufferers of this dreadful disorder? It was at that point that it became clear the extent to which we had all learned to pin our collective therapeutic hopes on the SGAs during the previous decade. After anger, came the phase of depression: where does this leave us and what do we do now? And finally, it seems, the phase of acceptance. In the end, it is perhaps striking how quickly this acceptance, that SGAs as a group of drugs are not such a major advance as once thought, has come about. Maybe we had a hunch this was so all along.
Stephan Leucht and colleagues have over recent years been in the vanguard of using meta-analysis to address key issues and controversies in antipsychotic drug treatment. Their recent article brings three of their most recent meta-analyses to bear on one of the most controversial of these questions in recent years: are the second-generation antipsychotic drugs (SGAs) actually no better than their much-maligned first-generation predecessors (FGAs)?
As we all know, the claims we came to believe in the 1990s were that the non-clozapine second-generation antipsychotic drugs (SGAs) were superior to first-generation antipsychotic drugs (FGAs) in terms of positive symptoms, negative symptoms, mood symptoms, adverse effects, adherence rates and neurocognitive effects. The increased acquisition costs, or price, of the new drugs were worth it in terms of the improved quality of life they delivered and, in any case, these costs would soon be recouped through savings on in-patient stays. These claims were readily taken up by a community of clinicians which had been startled by the unprecedented efficacy of clozapine (Kane et al. Reference Kane, Honigfeld, Singer and Meltzer1988) and were thus primed to accept the idea, which seemed revolutionary at the time, that there could be antipsychotic drugs with efficacy and safety profiles which differed from the conventional drugs. The SGAs, or new atypicals, were seen as a class because they all grew out of a new preclinical model, because they shared a reduced risk of extrapyramidal side-effects (EPS), and because they were marketed as such.
The strengths of Leucht and colleagues' (Reference Leucht, Kissling and Davis2009) meta-analyses is that they are constructed to test specific hypotheses about possible sources of error, such as supposed sources of bias, as well as examining the major questions about superiority in efficacy or safety. In one sense, there is a particular irony to this. It was the repeated deployment of meta-analyses which contributed to the body of evidence purporting to show the clear advantages of SGAs in terms of efficacy and safety, compared to their predecessors. However, there were straws in the wind as early as 2000, when Geddes and colleagues' (Reference Geddes, Freemantle, Harrison and Bebbington2000) meta-analysis suggested that a large part of the variance in the difference between SGAs and FGAs was down to the choice of the comparator drug and its dosage: specifically, haloperidol at rather high dosage. The rest is recent history. Starting with Rosenheck et al.'s (Reference Rosenheck, Perlick, Bingham, Liu-Mares, Collins, Warren, Leslie, Allan, Campbell, Caroff, Corwin, Davis, Douyon, Dunn, Evans, Frecska, Grabowski, Graeber, Herz, Kwon, Lawson, Mena, Sheikh, Smelson and Smith-Gamble2003) trial which showed no advantages to olanzapine compared to haloperidol when used in modest dosages with anticholinergic cover, through CATIE (Lieberman et al. Reference Lieberman, Stroup, McEvoy, Swartz, Rosenheck, Perkins, Keefe, Davis, Davis, Lebowitz, Severe and Hsiao2005), then CUtLASS (Jones et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006), then EUFEST (Kahn et al. Reference Kahn, Fleischhacker, Boter, Davidson, Vergouwe, Keet, Gheorghe, Rybakowski, Galderisi, Libiger, Hummer, Dollfus, Lopez-Ibor, Hranov, Gaebel, Peuskens, Lindefors, Riecher-Rossler and Grobbee2008), the received wisdom about the superiority of the SGAs has been robustly challenged.
One of the often-stated weaknesses of meta-analysis is that it is a hostage to fortune as regards the variable quality of the component trials. So, it is good to see that one of the strengths of the paper is that the authors examine systematically possible sources of bias in the individual trials examined. The issue of industry bias is dissected usefully to show that much of the reason why 90% of industry-funded trials find in favour of the company's compound is down to how respective authors choose to interpret (or spin) the data, rather than the data themselves being clearly in favour of the experimental drug. Another possible source of bias examined is in the choice of a poorly tolerated comparator drug. It emerges after all that there is little straight support for the assertion of the Geddes et al. (Reference Geddes, Freemantle, Harrison and Bebbington2000) meta-analysis that the effect size of the efficacy difference between SGAs and FGAs, specifically haloperidol, was due to high dosing of the comparator drug. Nonetheless, it remains true that most SGA–FGA comparisons have selected haloperidol as the comparator and one cannot help but speculate whether companies tended to choose this drug because of its known high rates of EPS. As the authors suggest, these trials may have thrown up different conclusions if a medium, rather than high, potency FGA comparator had been used. Linked to this is another alleged source of type I error, generating findings erroneously in support of the experimental compound, namely the practice of dealing with missing data-points from drop-outs by using the ‘last observation carried forward’ statistical adjustment. Leucht tested whether this was truly a source of error by reanalysing the original datasets from some trials and finding that it was not. Nonetheless, it would be surprising if this crude statistical technique did not generate bias in some studies, particularly where there were major disparities early on in drop-out rates between treatment arms. Trials such as CATIE and CUtLASS moved beyond this potentially misleading approach, using instead imputational techniques to deal with missing data.
Importantly and more generally, Leucht and colleagues point to widespread design limitations in antipsychotic drug trials. The absence of an agreed criterion for clinical response in schizophrenia contrasts with universally adopted Hamilton-score-based criteria for response in trials of major depression, for example. In schizophrenia trials, important clinical endpoints such as response, remission and relapse have no widely agreed operational definition, which will always hamper summary approaches such as meta-analysis.
Some of the conclusions of the authors might be open to debate. For instance, they suggest that a sufficiently dosed double-blind trial of clozapine versus other SGAs is still needed. Is it? CATIE 2 (McEvoy et al. Reference McEvoy, Lieberman, Stroup, Davis, Meltzer, Rosenheck, Swartz, Perkins, Keefe, Davis, Severe and Hsiao2006) and CUtLASS 2 (Lewis et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006), which were open-label comparisons of clozapine versus non-clozapine SGAs, used blinded raters to overcome the potential bias inherent to open trials. The two trials, although designed differently, showed similar outcomes: a statistically and clinically significant advantage to clozapine over other SGAs. Presumably the authors believe that this observed advantage still might plausibly arise from the non-specific therapeutic effect of regular clinical contact through the mandatory blood monitoring.
Where the authors, and other commentators too, get into a bit of a tangle is over the essentially semantic issue of ‘are SGAs a class?’ The authors conclude that SGAs are not a homogenous category and therefore not a class. As they rightly say, individual SGAs differ one from another in many properties, including efficacy, safety and pharmacology. However, the class of SGAs is explicitly defined by their name: second generation. As such, chronologically defined, they are clearly a class – they were discovered later than the FGAs – and it is precisely this agnosticism which has led to this name being preferred. The term it replaced, ‘atypical’, implied a class defined by either (it was never definitively agreed) a common preclinical mechanism (atypical pharmacology) or a common clinical profile (atypical safety profile). So, SGAs are evidently a class on semantic grounds, but whether it is useful in practice to classify them so, in contrast to FGAs, depends on why one might wish to do it. Shortly after the Leucht et al. article was written, the UK clinical guidelines body, NICE, published its 400-page 2009 update of schizophrenia treatment guidelines supported by a range of systematic reviews. At no stage in the actual guideline is the distinction between FGAs and SGAs employed, and even for first-line treatment there is no explicit recommendation that SGAs are preferred. In considering relapse prevention for example, the guideline states:
All the antipsychotics identified for review have established supremacy over placebo in preventing relapse, although the evidence that any individual drug or group of drugs (FGA v. SGA) has greater efficacy or tolerability is still very uncertain … Any small advantage (offered by SGAs) of reduced EPS may be offset by other adverse consequences not shown by earlier drugs.
The NICE guideline states at the outset that ‘Issues of … affordability are determined by the NHS’. The main distinction at the policy level between FGAs and SGAs has been their acquisition cost. FGAs are cheap, and SGAs less so by at least an order of magnitude. Of course, there are caveats here. The overall costs of health and social care for someone with schizophrenia is high, and drug costs, even for the relatively expensive SGAs, turn out to be a small proportion of this. Moreover, SGAs are one by one becoming generic and, although it will take several years for the price of such generics to approach that of the old drugs, this will blur the usefulness of a distinction on cost. Nonetheless, Leucht and colleagues downplay the issues of cost effectiveness. It is important to realize that both CATIE (Rosenheck et al. Reference Rosenheck, Leslie, Sindelar, Miller, Lin, Stroup, McEvoy, Davis, Keefe, Swartz, Perkins, Hsiao and Lieberman2006) and CUtLASS (Davies et al. Reference Davies, Lewis, Jones, Barnes, Gaughran, Hayhurst, Markwick and Lloyd2007) involved sophisticated cost-effectiveness analyses because the rationale for both studies included issues of relative cost. The initial letters of the CUtLASS acronym stand for ‘cost utility’. Analyses of both studies found FGAs to be the ‘dominant choice’: that is, no less effective, but cheaper. In their conclusion to this section, the authors state that ‘Some may suggest that the money spent on the SGAs should rather be spent on psychosocial therapies. But, unfortunately, the money might be allocated to other areas of medicine such as cardiology, or cancer.’ This could be true, but surely it would be seen as a miserable failure of academic leadership if we were not able to ensure at a national and local policy level that cost savings made from switching to cheaper drugs with equal effectiveness were not reinvested in other evidence-based treatments for patients with schizophrenia.
Looking back over the past 4 or 5 years at the reaction of the field to the data-driven reappraisal of the relative place of SGAs, it is perhaps not too whimsical to see that it has followed the stereotypic phases of a bereavement reaction, and those at the sharp end of the disclosure and discussion of the results of CATIE and CUtLASS can attest to this. First, there was a denial phase: the results were dismissed as due to poor trial design and type II statistical error. Next, there followed a phase of anger. Why were these researchers promulgating such damaging nonsense, which threatened to set the field back 20 years? Most difficult was when this reaction came not from companies, who were usually restrained, nor clinicians, who often were not, but from patient groups. Shortly after the publication and attendant press releases of the CUtLASS 1 trial, the study team took a call from the chief executive of a UK mental health charity. What were we doing, she asked, destroying the hope of better treatment for sufferers of this dreadful disorder? It was at that point that it became clear the extent to which we had all learned to pin our collective therapeutic hopes on the SGAs during the previous decade. After anger, came the phase of depression: where does this leave us and what do we do now? And finally, it seems, the phase of acceptance. In the end, it is perhaps striking how quickly this acceptance, that SGAs as a group of drugs are not such a major advance as once thought, has come about. Maybe we had a hunch this was so all along.
Declaration of Interest
S.L. has received speaker fees and travel honoraria from the following pharmaceutical companies: AstraZeneca, Janssen-Cilag, Lilly, Bristol-Myers Squibb, Pfizer.