Evidence-based practice (EBP) is rapidly becoming the required standard for practicing mental health clinicians, with training, credentialing and funding increasingly being linked to its use (Tanenbaum, Reference Tanenbaum2005; Magnabosco, Reference Magnabosco2006; Isett et al. Reference Isett, Burnam, Coleman-Beattie, Hyde, Morrissey, Magnabosco, Rapp, Ganju and Goldman2008; Raghavan et al. Reference Raghavan, Bright and Shadoin2008; Cooper & Aratani, Reference Cooper and Aratani2009; Slomski, Reference Slomski2012). Hence, it behooves the mental health field to examine how clinicians are to obtain the accurate, up-to-date knowledge of effectiveness research they are being asked to incorporate into their treatment decisions.
Practitioners have repeatedly reported that the time and methodological expertise required to keep abreast of the latest journal articles is prohibitive given their daily clinical demands (Armstrong et al. Reference Armstrong, Waters, Crockett and Keleher2007; Forsner et al. Reference Forsner, Hansson, Brommels, Wistedt and Forsell2010; Hannes et al. Reference Hannes, Pieters, Goedhuys and Aertgeerts2010; Gallo & Barlow, Reference Gallo and Barlow2012). Instead, they have come to rely increasingly on systematic reviews and evidence-based treatment guidelines generated from such reviews to gather the best available evidence and to distill it into useful recommendations for them (APA Council of Representatives, 2005; Littell, Reference Littell2008; Ahmad et al. Reference Ahmad, Boutron, Deschartres, Durieux and Ravaud2010). In recent years, a plethora of systematic reviews, guidelines and evidence-based treatment lists have arisen (APA Task Force on Evidence-Based Practice for Children and Adolescents, 2008; Stiles et al. Reference Stiles, Boothroyd, Dhont, Beiler and Green2009). Yet, clinicians have voiced doubts both about the credibility of such sources and their applicability to the types of cases seen in daily practice (Pagoto et al. Reference Pagoto, Spring, Coups, Mulvaney, Coutu and Ozakinci2007; Nelson & Steele, Reference Nelson and Steele2008; Forsner et al. Reference Forsner, Hansson, Brommels, Wistedt and Forsell2010; Hannes et al. Reference Hannes, Pieters, Goedhuys and Aertgeerts2010). Studies indicating significant differences in treatment recommendations among evidence-based guidelines (Gaebel et al. Reference Gaebel, Weinmann, Sartorius, Rutz and McIntyre2005; Forbes et al. Reference Forbes, Creamer, Bisson, Cohen, Crow, Foa, Friedman, Keane, Kudler and Ursano2010; Vasse et al. Reference Vasse, Vernooij-Dassen, Cantegreil, Franco, Dorenlots, Woods and Moniz-Cook2012), as well as indications of publication bias (Thase, Reference Thase2008; Matthew & Charney, Reference Matthew and Charney2009), may have contributed to such perceptions.
The mental health field would benefit from having a definitive evidence-based review source, respected for its objectivity and methodological rigour, which can be either accessed directly by practitioners or used as a source of guideline development (Barbui & Tansella, Reference Barbui and Tansella2011). The Cochrane Systematic Review System, with over 5000 healthcare reviews (Cochrane Library, 2012), rises as the premier candidate for such a post. Cochrane reviews have been found to be more methodologically rigorous than other systematic reviews (Jørgensen et al. Reference Jørgensen, Hilden and Gøtzsche2006; Moseley et al. Reference Moseley, Elkins, Herbert, Maher and Sherrington2009) and are seen as highly credible by health care providers (Rosenbaum et al. Reference Rosenbaum, Glenton and Cracknell2008).
Currently, the Cochrane Collaboration is composed of 53 relatively autonomous review groups (CRGs), six of which directly focus on mental health or substance misuse (depression, anxiety and neurosis; schizophrenia; developmental, psychosocial and learning problems; dementia and cognitive impairment; drugs and alcohol; and tobacco addiction). As Cochrane reviews are authored by volunteers, author interest and a CRG's agenda historically have factored heavily in determining which topics will be prioritized for review, although the prioritization process differs by CRG (Grimshaw, Reference Grimshaw2004; Nasser et al. Reference Nasser, Welch, Tugwell, Ueffing, Doyle and Waters2012). Some have questioned if such a process yields adequate coverage of topics important for other stakeholders (i.e., practitioners, patients, policy makers, etc.) (Ahmad et al. Reference Ahmad, Boutron, Deschartres, Durieux and Ravaud2010; Gill et al. Reference Gill, Wang, Mant, Hartling, Heneghan, Perera, Klassen and Harden2011). This study will examine the Cochrane Collaboration's mental health and substance misuse entries to explore its breadth of coverage in these areas.
Methods
Two methods were used to extract mental health and substance related entries from the 2005 to April 2012 Cochrane Systemic Reviews Database. First, all the entries for each CRG were assessed as to whether or not they related to a mental health or substance misuse topic based on the entry's title. Second, the database was searched for 196 mental health and substance related keywords (e.g. mood, anxiety, dyslexia, delinquency, amphetamines, etc.) to locate any additional relevant entries. In cases where the title was ambiguous, an examination of the entry's objectives, type of participants and types of intervention sections was employed to classify the entry. A total of 1019 mental health and substance misuse entries were obtained through these two methods.
Each entry was characterized as to its current status (protocol or review) and the CRG from which it originated. Entries were also coded as to diagnostic topic area and type of intervention. The diagnostic topic was coded primarily based on the entry's title, but in cases where the title was ambiguous, examination of the entry's type of population and types of intervention sections was used for clarification. The 64 diagnostic topic areas (see Table 1) are based on categories in the DSM-IV-TR (American Psychiatric Association, 2000). With respect to type, interventions were characterized as involving medication (e.g. psychopharmacological drugs, Chinese herbal medicines, St John's Wart, etc.), psychotherapy/counselling (e.g. cognitive behavioural therapy, family therapy, support groups, etc.), other (e.g. acupuncture, exercise, occupational therapy, electro-convulsive therapy (ECT), etc.) and the different combinations of these three categories.
Table 1. Frequency of diagnostic groups by protocol and type of intervention (n = 1019) Diagnostic topic
*Only elimination disorders affecting youth were included in this study.
Results
Topic coverage
A total of 1019 entries focused on mental health or substance misuse. One out of every five entries (20.0%) focused on serious mental illness/psychosis (see Table 1). Substance misuse represented the next most common diagnostic topic, representing 16.5% of all entries. However, coverage within substance was uneven with over a third (n = 61) of all the substance entries dedicated to smoking, twice the number focused on alcohol or opiates. Cognitive impairment was the third most popular topic, with one of out every eight (12.6%) entries addressing cognitive impairments. The majority of these were dementia interventions.
Nearly 12% (11.6%) of the entries targeted mood disorders. Relatively few of the mood entries, however, concentrated specifically on bipolar disorder (n = 18). The number of anxiety disorder entries was less than half of those seen for mood disorders (5.5%), with a quarter of all anxiety reviews/protocols (n = 14) addressing anxiety disorders within the context of a medical situation (e.g. dental anxiety, preoperative anxiety, etc.). No other diagnostic group exceeded 5% of the sample.
Of particular note, entries focused on dually diagnosed or comorbid populations represented only 4.6% of the sample, which is significantly less than their percentage in typical clinical practice (Kessler et al. Reference Kessler, Merikangas, Wang, Levin, Hennessy and Petrila2010; Einfeld et al. Reference Einfeld, Ellis and Emerson2011). Somatic disorders also composed 4.6% of the sample, however, it is worth noting that relatively few entries in this category (n = 7) reflected traditional DSM-IV-TR somatoform disorders, with the majority of this group instead being composed of disorders whose classification as somatoform is controversial (e.g. chronic fatigue, fibromyalgia, etc.) (Brown, Reference Brown2007). A number of disorders with onsets in infancy, childhood and adolescence were more sparsely represented in the Cochrane database. Autism spectrum, attention deficit hyperactivity disorder (ADHD), tic disorders, enuresis/encopresis, externalizing disorders, child maltreatment, intellectual/learning disabilities, speech disorders and developmental motor delays combined represented only 10% (10.4%) of the mental health and substance misuse sample.
Types of intervention
On average 57.3% of the entries for mental health and substance misuse focused, all or in part, on interventions involving medication; in contrast, less than a third (29.3%) of mental health and substance misuse entries were about psychotherapy, counselling or other non-medication interventions. The greater emphasis on pharmacological interventions may reflect the Cochrane Collaboration's preference for using randomized clinical trials when conducting systematic reviews (Higgins & Green, Reference Higgins and Green2011). Interestingly, medication interventions were significantly less prevalent (32.1%) in disorders known for childhood-onset (autism spectrum, ADHD, externalizing disorders, child maltreatment, intellectual/learning disabilities, speech and developmental motor delays) than in other disorders (60.2%) (χ2 (3, N = 1019) = 30.80, p < 0.001, Ф = 0.174).
Protocols v. reviews
Of the 1019 mental health and substance misuse entries, 698 (68.5%) were reviews and 321 (31.5%) were protocols. In the Cochrane Collaboration a protocol on a topic is first published and then converted into a full systematic review within 2 years (Higgins & Green, Reference Higgins and Green2011), although concerns have been raised about the ability of the Cochrane Collaboration to adhere to this timeline (French et al. Reference French, McDonald, McKenzie and Green2005; Bow et al. Reference Bow, Klassen, Chisholm, Tiosvold, Thomson, Klassen, Moher and Hartling2010). The review, not the protocol, has the potential to provide clinicians with treatment recommendations. Protocols composed the majority (≥75%) of the entries in the suicide/self-injury, impulse control disorders and personality disorder areas, indicating that despite the number of entries as yet relatively little treatment guidance is being provided on these topics to those making health care decisions.
Cochrane review groups
Diagnostic groups also differed in terms of the number of CRGs involved in producing protocols/reviews. Eating disorders, impulse control disorders, autism spectrum, ADHD, personality disorders, tic disorders and general emotional/adjustment disorders had all their entries arise from a single CRG. At the other end of the spectrum, protocols and reviews concerning sleep disorders were spread across 14 different CRGs, with anxiety, mood and substance dispersed among 9 to 11 CRGs each. Although the Cochrane system dedicates resources to CRG integration (Cochrane Collaboration, 2012), such an arrangement increases the odds of duplication of effort such as in 2007 when two separate systematic reviews were published entitled ‘Psychosocial and psychological interventions for treatment of postpartum depression’ (from the depression, anxiety and neurosis group) and ‘Psychosocial and psychological interventions for treating antenatal depression’ (from the pregnancy and childbirth group). In addition to the issue of duplication of effort, there can be difficulty in providing comprehensive guidance for clinicians when the responsibility for the issue is distributed in such a fragmented manner.
Discussion
EBP requires clinicians to understand the current research on treatment efficacy (Tanenbaum, Reference Tanenbaum2005; Magnabosco, Reference Magnabosco2006; Isett et al. Reference Isett, Burnam, Coleman-Beattie, Hyde, Morrissey, Magnabosco, Rapp, Ganju and Goldman2008; Raghavan et al. Reference Raghavan, Bright and Shadoin2008; Cooper & Aratani, Reference Cooper and Aratani2009; Slomski, Reference Slomski2012). Many clinicians obtain that knowledge, directly or through guidelines, from conclusions and recommendations derived from systematic reviews (APA Council of Representatives, 2005; Littell, Reference Littell2008; Ahmad et al. Reference Ahmad, Boutron, Deschartres, Durieux and Ravaud2010). However, clinicians have expressed doubts about both the applicability and credibility of such sources, perhaps fostered by guidelines purporting to represent the same research base but yet espousing contradictory treatment recommendations (Gaebel et al. Reference Gaebel, Weinmann, Sartorius, Rutz and McIntyre2005; Pagoto et al. Reference Pagoto, Spring, Coups, Mulvaney, Coutu and Ozakinci2007; Nelson & Steele, Reference Nelson and Steele2008; Forbes et al. Reference Forbes, Creamer, Bisson, Cohen, Crow, Foa, Friedman, Keane, Kudler and Ursano2010; Forsner et al. Reference Forsner, Hansson, Brommels, Wistedt and Forsell2010; Hannes et al. Reference Hannes, Pieters, Goedhuys and Aertgeerts2010; Vasse et al. Reference Vasse, Vernooij-Dassen, Cantegreil, Franco, Dorenlots, Woods and Moniz-Cook2012). Having a single systematic review source with an unquestionably high level of credibility may be beneficial to the mental health and substance misuse fields. The Cochrane Collaboration possesses the credibility to legitimately become that definitive review source (Rosenbaum et al. Reference Rosenbaum, Glenton and Cracknell2008); however, it is unclear if its mental health and substance abuse coverage is sufficiently broad enough for it to be used as such.
Examination for all mental health and substance related entries in the 2005 to April 2012 Cochrane database revealed a large number of such entries (N = 1019). The majority (60.6%) of the mental health and substance misuse entries were in the areas of serious mental illness, substance (primarily smoking, alcohol and opiates), mood disorders and cognitive impairments (particularly dementia). The daily and economic burdens posed by these diagnoses are unquestionably high (World Health Organization, 2008; Luengo-Fernandez et al. Reference Luengo-Fernandez, Leal and Gray2010); yet other psychiatric disorders with less coverage are as frequent and have been found to have comparable levels of burden (Kessler et al. Reference Kessler, Merikangas, Wang, Levin, Hennessy and Petrila2010; Alonso et al. Reference Alonso, Petukhova, Vilagut, Chatterji, Heeringa, Üstün, Alhamzawi, Viana, Angermeyer, Bromet, Bruffaerts, de Girolamo, Florescu, Gureje, Haro, Hinkov, Hu, Karam, Kovess, Levinson, Medina-Mora, Nakamura, Ormel, Posada-Villa, Sagar, Scott, Tsang, Williams and Kessler2011; Wittchen et al. Reference Wittchen, Jacobi, Rehm, Gustavsson, Svensson, Jönsson, Olesen, Allgulander, Alonso, Faravelli, Fratiglioni, Jennum, Lieb, Maercker, van Os, Preisig, Salvador-Carulla, Simon and Steinhausen2011). Coverage was noticeably thinner with comorbid conditions and various disorders whose onset typically is in infancy, childhood or adolescence. The fact that some areas (personality disorders, suicide/self-injury and impulse control disorders) are dominated by protocols, v. reviews, may give the illusion that there is greater guidance available in that area than is actually present at the moment, although that situation should be largely addressed through the passage of time as protocols are converted to reviews.
It is to be noted that the majority of Cochrane entries focused on interventions involving medication (57.3%), although this rate is significantly lower for a group of disorders characterized by a youth-onset. Approximately half as many entries were directed towards psychotherapy, counselling or other types of non-medication intervention as compared with interventions involving medication. The strong focus towards medication interventions may make the Cochrane Collaboration somewhat less useful for mental health providers without prescription privileges.
Last, the coverage of certain diagnostic areas was spread across a large number of CRGs. For the Cochrane system to be truly useful to practicing clinicians it must thoughtfully allocate its resources so as to provide maximum coverage to topics most likely to be faced in clinical practice, neither duplicating efforts nor leaving important areas uncovered. This may be a difficult goal to achieve if coverage for a topic is distributed across a large number of CRGs.
These findings point to the importance of prioritization of review topics. Historically, the Cochrane Collaboration has embraced a ‘bottom-up’ structure, where the author and CRG interest primarily set review priorities (Nasser et al. Reference Nasser, Welch, Tugwell, Ueffing, Doyle and Waters2012). Hence, the amount of coverage that has been devoted to a given topic could reflect a variety of factors, ranging from author interest level to the breadth of a given CRG's topic list to editorial openness towards inclusion of non-randomized studies or the publication of empty reviews (which do not have any studies meeting inclusion criteria) (Yaffe et al. Reference Yaffe, Montgomery, Hopewell and Sheppard2012).
Yet, given its rapid growth, current size and growing role, such a guiding framework may no longer be the best fit. Consistent with this, the Cochrane Collaboration has been urged to establish a transparent system for prioritizing reviews to better meet the needs of its users (Purgato et al. Reference Purgato, Barbui and Adams2011; Nasser et al. Reference Nasser, Welch, Tugwell, Ueffing, Doyle and Waters2012). The Cochrane Collaboration recently identified establishing a priority setting system as a strategic recommendation for the organization (MacLehose et al. Reference MacLehose, Hilton and Tovey2012). In 2008, it began funding several pilot prioritization projects (i.e., using practice guidelines to determine review priorities, patient-professional partnerships such as the James Lind Alliance, prioritizing know-do gaps in low and middle income countries, etc.) (Cochrane Agenda and Priority Setting Methods Group, 2012). These and other projects have yielded data on various priority setting systems (Purgato et al. Reference Purgato, Barbui and Adams2011; Wale et al. Reference Wale, Belizan, Nadel, Jeffrey and Vij2011; Handoll et al. Reference Handoll, Stott, Elstub, Elliott, Kavanagh and Madhok2012), but as yet there is no uniformity among the CRGs in terms of whether they engage in prioritizing review topics and, if so, the method used (Nasser et al. Reference Nasser, Welch, Tugwell, Ueffing, Doyle and Waters2012). The challenges in implementing a ‘top-down’ priority system in a system where the reviews are completed by volunteer researchers are considerable (e.g. eliciting interest, ensuring expertise in the priority topic, etc). Maximizing author incentives (Tovey, Reference Tovey2010) will be important in order to effect such an organizational change.
Several limitations are important to note with regard to these findings. Classification of an entry as mental health or substance misuse, as well as classification of type of intervention, was done primarily by the information given in each entry's title. When classification was unclear based on title the appropriate section of the protocol or review was accessed to obtain clarity; however, it is possible that seemingly clear titles may not have accurately conveyed the essence of the entry resulting in misclassifications. In addition, typically a limited number of entries (>5%) are subsequently withdrawn from the Cochrane database. As information regarding withdrawal status was not factored in, it is possible that the number of mental health and substance misuse studies in the sample is a slight over-estimate of the true number of non-withdrawn entries. Last, for systematic reviews to be truly useful to practicing clinicians they must also be accessible, reflective of the current literature, provide definitive guidance with regard to treatment options providers can use and match the populations clinicians typically see (El Dib et al. Reference El Dib, Atallah and Andriolo2007; Moher et al. Reference Moher, Tsertsvadze, Tricco, Eccles, Grimshaw, Sampson and Barrowman2007; Rosen & Noach, Reference Rosen and Noach2010; Tricco et al. Reference Tricco, Tetzlaff and Moher2011; Armstrong et al. Reference Armstrong, Pettman, Burford, Doyle and Waters2012; Yaffe et al. Reference Yaffe, Montgomery, Hopewell and Sheppard2012). The current study only examines whether the diagnostic topic areas were addressed as an entry in the Cochrane database, not whether it does or does not meet these other criteria that would assist in making it optimally useful to practicing clinicians.
Conclusions
Although the Cochrane Systemic Review system is clearly growing in its mental health and substance misuse coverage, as yet its most comprehensive guidance is found for medication interventions for several prevalent disorders with high burden profiles (serious mental illness/psychosis, substance, dementia and depression). Although information is available regarding other diagnoses and therapeutic interventions, it is unevenly distributed and more limited. Use of a more centralized, directed approach to broaden its diagnostic coverage and diversify types of interventions reviewed would increase Cochrane's relevance for those seeking evidence-based guidance when providing mental health and substance misuse services.
Declaration of interest
No competing interests exist. No economic support was received.
