In their Special Article, Cristea & Naudet (Reference Cristea and Naudet2017) report the results of a citation content analysis on the paper ‘Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses’ published by Leucht et al. (Reference Leucht, Hierl, Kissling, Dold and Davis2012) in the British Journal of Psychiatry. This meta-analysis, which has become an influential paper being cited 135 times since its publication, aimed to put the efficacy of psychotropic drugs into the perspective of that of major medical drugs. The conclusions by Leucht et al. are that the psychiatric drugs are not less efficacious than other drugs, but their efficacy should be considered in the context of several factors, such as disease's seriousness, suffering induced, natural course, duration, outcomes, adverse events and societal values.
Cristea and Naudet argue that this simple and straightforward message has been misinterpreted. In their citation content analysis, the authors show that the original paper was overwhelmingly uncritically received. In fact, most of the citing papers have not reported all caveats that Leucht et al. mentioned in the limitations’ section of their manuscript. Therefore, the citation flow of this meta-analysis would represent an example of ‘citation bias’, a phenomenon not new in the scientific literature, in which data coming from an original paper are mistakenly reported, causing a misrepresentation of trusts and expectations (Tatsioni et al. Reference Tatsioni, Bonitsis and Ioannidis2007; Greenberg, Reference Greenberg2009). Mogull (Reference Mogull2017) found that quotation errors are present not only in psychiatry, but affect all branches of medicine, with an incidence of 15%; in particular, 65% of these are major content errors (i.e., the cited reference is unrelated with or contradicts the original finding).
The citation bias can be due to the fact that data are used or interpreted differently by investigators depending on their personal beliefs, scientific theories or personal interests (Resch et al. Reference Resch, Ernst and Garrow2000). In particular, citation bias can be due to the intentional inclusion of inappropriate citations and can reflect the authors’ intention to increase the citation counts of unrelated articles (Mogull, Reference Mogull2017); in some other cases, the citation bias can be just the results of authors’ lack of accuracy in reporting previous findings (Jergas & Baethge, Reference Jergas and Baethge2015).
The consequences of citation bias can be detrimental to the scientific process of improving knowledge, since it can contribute to create false authorities (i.e., if a paper is highly cited, it implies that it is relevant for the scientific community) or to develop a false evidence that can have an impact on clinical practice, although it is not supported by the original findings. In particular, the point raised by Cristea and Naudet has clear practical implications, since false beliefs introduced through quotations’ manipulation can negatively influence research, policies and clinical practice. In fact, policy makers can be erroneously induced to promote health interventions that are not supported by scientific data, clinicians may be inclined to modify their clinical practice, and researchers may propose protocols without a solid evidence base.
The second important issue raised by Cristea and Naudet in their Special Article refers to the existing gap between efficacy and effectiveness of psychiatric pharmacological treatments. We agree with the authors that ‘An effect size is not reducible to just numbers and the very idea of small, medium or large effect…is intricately interwoven with the type of outcome’. This is what has been omitted too often when the review by Leucht et al. has been quoted. We think that the point raised by Cristea and Naudet is not whether psychotropic drugs are effective or not; rather, the authors have stressed the fact that the effect size of a drug should be considered in light of several caveats, that have been reported by Leucht et al. but that have been omitted by the citing papers. This is a very important issue when considering the role of medications, especially in psychiatric practice. In fact, it happens quite often that a given (pharmacological or psychosocial) treatment, which has proven to be effective in randomised clinical trials, fails to produce the same effects when applied in routine settings (Depp & Lebowitz, Reference Depp and Lebowitz2007; Fiorillo et al. Reference Fiorillo, Del Vecchio, Luciano, Sampogna, De Rosa, Malangone, Volpe, Bardicchia, Ciampini, Crocamo, Iapichino, Lampis, Moroni, Orlandi, Piselli, Pompili, Veltro, Carrà and Maj2015). The effects of a psychotropic drug in randomised controlled trials (namely efficacy) should be verified in routine clinical conditions (namely effectiveness), and in light of several mediators – such as timing, dosing, receptors binding and patients’ compliance to medications – and moderators, such as gender, age, presence of comorbidities or side effects, which can alter the efficacy of a given drug. A small or modest variation on a Likert scale can be associated with a small effect size, but also with a significant improvement of patients’ levels of functioning, quality of life and other clinical and social outcomes, which are not detectable by effect sizes. However, we want to highlight an important message of the paper by Leucht et al.: despite all its limitations, there are no doubts that the efficacy of psychotropic drugs is supported by randomised controlled trials. Data coming from well-designed studies should always be interpreted with critical thinking in order to be translated into clinical practice. We would need to develop a culture that is not biased by ideological prejudices, a priori theories and/or researchers’ personal beliefs (Maj, Reference Maj2016). The importance of the therapeutic alliance should be highlighted in psychiatric education and practice; information about psychotropic drugs should be balanced, unambiguous and unconditioned; the psychological and cultural barriers that limit the use of psychotropic drugs and reduce patients’ adherence to treatments shall be removed.
In conclusion, the paper by Cristea and Naudet is very welcome, as it sheds light on a neglected topic in mental health research, that of correctly citing scientific papers. Moreover, we believe that it will help the discussion on the efficacy and effectiveness of psychotropic drugs, with the hope that the debate will be unambiguous and unconditioned, but driven by research findings.
In their Special Article, Cristea & Naudet (Reference Cristea and Naudet2017) report the results of a citation content analysis on the paper ‘Putting the efficacy of psychiatric and general medicine medication into perspective: review of meta-analyses’ published by Leucht et al. (Reference Leucht, Hierl, Kissling, Dold and Davis2012) in the British Journal of Psychiatry. This meta-analysis, which has become an influential paper being cited 135 times since its publication, aimed to put the efficacy of psychotropic drugs into the perspective of that of major medical drugs. The conclusions by Leucht et al. are that the psychiatric drugs are not less efficacious than other drugs, but their efficacy should be considered in the context of several factors, such as disease's seriousness, suffering induced, natural course, duration, outcomes, adverse events and societal values.
Cristea and Naudet argue that this simple and straightforward message has been misinterpreted. In their citation content analysis, the authors show that the original paper was overwhelmingly uncritically received. In fact, most of the citing papers have not reported all caveats that Leucht et al. mentioned in the limitations’ section of their manuscript. Therefore, the citation flow of this meta-analysis would represent an example of ‘citation bias’, a phenomenon not new in the scientific literature, in which data coming from an original paper are mistakenly reported, causing a misrepresentation of trusts and expectations (Tatsioni et al. Reference Tatsioni, Bonitsis and Ioannidis2007; Greenberg, Reference Greenberg2009). Mogull (Reference Mogull2017) found that quotation errors are present not only in psychiatry, but affect all branches of medicine, with an incidence of 15%; in particular, 65% of these are major content errors (i.e., the cited reference is unrelated with or contradicts the original finding).
The citation bias can be due to the fact that data are used or interpreted differently by investigators depending on their personal beliefs, scientific theories or personal interests (Resch et al. Reference Resch, Ernst and Garrow2000). In particular, citation bias can be due to the intentional inclusion of inappropriate citations and can reflect the authors’ intention to increase the citation counts of unrelated articles (Mogull, Reference Mogull2017); in some other cases, the citation bias can be just the results of authors’ lack of accuracy in reporting previous findings (Jergas & Baethge, Reference Jergas and Baethge2015).
The consequences of citation bias can be detrimental to the scientific process of improving knowledge, since it can contribute to create false authorities (i.e., if a paper is highly cited, it implies that it is relevant for the scientific community) or to develop a false evidence that can have an impact on clinical practice, although it is not supported by the original findings. In particular, the point raised by Cristea and Naudet has clear practical implications, since false beliefs introduced through quotations’ manipulation can negatively influence research, policies and clinical practice. In fact, policy makers can be erroneously induced to promote health interventions that are not supported by scientific data, clinicians may be inclined to modify their clinical practice, and researchers may propose protocols without a solid evidence base.
The second important issue raised by Cristea and Naudet in their Special Article refers to the existing gap between efficacy and effectiveness of psychiatric pharmacological treatments. We agree with the authors that ‘An effect size is not reducible to just numbers and the very idea of small, medium or large effect…is intricately interwoven with the type of outcome’. This is what has been omitted too often when the review by Leucht et al. has been quoted. We think that the point raised by Cristea and Naudet is not whether psychotropic drugs are effective or not; rather, the authors have stressed the fact that the effect size of a drug should be considered in light of several caveats, that have been reported by Leucht et al. but that have been omitted by the citing papers. This is a very important issue when considering the role of medications, especially in psychiatric practice. In fact, it happens quite often that a given (pharmacological or psychosocial) treatment, which has proven to be effective in randomised clinical trials, fails to produce the same effects when applied in routine settings (Depp & Lebowitz, Reference Depp and Lebowitz2007; Fiorillo et al. Reference Fiorillo, Del Vecchio, Luciano, Sampogna, De Rosa, Malangone, Volpe, Bardicchia, Ciampini, Crocamo, Iapichino, Lampis, Moroni, Orlandi, Piselli, Pompili, Veltro, Carrà and Maj2015). The effects of a psychotropic drug in randomised controlled trials (namely efficacy) should be verified in routine clinical conditions (namely effectiveness), and in light of several mediators – such as timing, dosing, receptors binding and patients’ compliance to medications – and moderators, such as gender, age, presence of comorbidities or side effects, which can alter the efficacy of a given drug. A small or modest variation on a Likert scale can be associated with a small effect size, but also with a significant improvement of patients’ levels of functioning, quality of life and other clinical and social outcomes, which are not detectable by effect sizes. However, we want to highlight an important message of the paper by Leucht et al.: despite all its limitations, there are no doubts that the efficacy of psychotropic drugs is supported by randomised controlled trials. Data coming from well-designed studies should always be interpreted with critical thinking in order to be translated into clinical practice. We would need to develop a culture that is not biased by ideological prejudices, a priori theories and/or researchers’ personal beliefs (Maj, Reference Maj2016). The importance of the therapeutic alliance should be highlighted in psychiatric education and practice; information about psychotropic drugs should be balanced, unambiguous and unconditioned; the psychological and cultural barriers that limit the use of psychotropic drugs and reduce patients’ adherence to treatments shall be removed.
In conclusion, the paper by Cristea and Naudet is very welcome, as it sheds light on a neglected topic in mental health research, that of correctly citing scientific papers. Moreover, we believe that it will help the discussion on the efficacy and effectiveness of psychotropic drugs, with the hope that the debate will be unambiguous and unconditioned, but driven by research findings.