Introduction
It is estimated that over 1.9 billion people have access to over 312 million sites on the internet (Internet World Statistics, 2011; Netcraft, 2011) and that as many as 80% of internet users in developed countries use the internet to search for information on health problems, symptoms, diseases and treatments (Fox, Reference Fox2006; Kummervold et al. Reference Kummervold, Chronaki, Lausen, Prokosch, Rasmussen, Santana, Staniszewski and Wangberg2008). Information on mental disorders is commonly accessed online, particularly by those with a psychiatric diagnosis and their supporters or carers (Powell & Clarke, Reference Powell and Clarke2006; Ybarra & Suman, Reference Ybarra and Suman2006; Khazaal et al. Reference Khazaal, Chatton, Cochand, Hoch, Khankarli, Khan and Zullino2008).
The growth in health information on the internet has been followed by an increase in the number of studies analysing its quality. A review published in 2002 found that 55 of 79 such studies considered quality to be a problem, although accuracy varied between health domains, with up to 90% of diet and nutrition information assessed as being unreliable compared to only 5% of that for cancer (Eysenbach et al. Reference Eysenbach, Powell, Kuss and Sa2002). A recent review of studies assessing the quality of websites providing information about mental disorders found that most of the research concluded that quality was poor, although site selection and rating methods varied, with some having unknown validity (Reavley & Jorm, Reference Reavley and Jorm2011).
A relatively recent feature of the debate about the quality of health information on the internet centres on websites that involve users in information sharing and collaboration, rather than viewing them as passive consumers of content created by experts. Often known as ‘Web 2.0’, this participatory model of web usage is associated with numerous applications, including social networking sites, blogs, media-sharing sites and wikis. One of the best known of these is Wikipedia, the online encyclopaedia that anyone can edit. More than 50% of internet users now source information from Wikipedia (Zickhur & Rainie, Reference Zickhur and Rainie2011), which has over 3.3 million English-language articles and has become a prominent source of online health information (Laurent & Vickers, Reference Laurent and Vickers2009). In 2005, a study comparing the quality of science articles in Encyclopaedia Britannica with those in Wikipedia found numerous errors in both, but that the difference in accuracy was not particularly great (Giles, Reference Giles2005). Several other studies have explored the quality of health information on Wikipedia, with some concluding that the information was of poor quality, and others reporting that it was of acceptable or even high quality (Heilman et al. Reference Heilman, Kemmann, Bonert, Chatterjee, Ragar, Beards, Iberri, Harvey, Thomas, Stomp, Martone, Lodge, Vondracek, de Wolff, Liber, Grover, Vickers, Mesko and Laurent2011).
In this study, we explored the quality of the user-contributed mental health-related information on Wikipedia and compared this with information from sources that are centrally controlled, including websites, in addition to Encyclopaedia Britannica and a comprehensive psychiatry textbook. We examined systematically the quality of information on both a high-prevalence mental disorder (depression) and a low-prevalence severe disorder (schizophrenia).
Method
Selection of sites and topics
The selection of websites from which material was extracted was based on the top 10 Google search results for either of the terms ‘depression’ (in March 2010) or ‘schizophrenia’ (in May 2010). The sites chosen by this method are likely to reflect those encountered by a typical user (Eysenbach & Kohler, Reference Eysenbach and Kohler2002). Websites that were portals to the content of other sites were excluded. Six sites appeared in the top 10 search results for both topics, four were unique to depression and four to schizophrenia. Overall, 14 websites were selected.
Ten mental health-related topics were chosen, five relating to depression and five to schizophrenia. An attempt was made to choose topics that were relatively specific (to facilitate ease of searching), rapidly evolving (to facilitate assessment of up-to-dateness) and controversial (to facilitate assessment of accuracy and breadth of coverage). The depression topics were: (1) antidepressants and suicide in young people; (2) gambling and depression; (3) side-effects of electroconvulsive therapy (ECT) and depression; (4) fish oils for depression; and (5) the relationship between attention deficit hyperactivity disorder (ADHD) and depression. The schizophrenia topics were: (1) the relationship between cannabis and psychosis/schizophrenia; (2) childhood onset of psychosis; (3) schizophrenia and violence; (4) side-effects of antipsychotics; and (5) stigma and schizophrenia.
Using the topic terms (or synonyms) as key words for the searches or through manual browsing, content relating to these topics was extracted from the selected websites and also from the most recent edition of Kaplan & Sadock's Comprehensive Textbook of Psychiatry (Sadock et al. Reference Sadock, Sadock, Ruiz and Kaplan2009) and the online version of Encyclopaedia Britannica. Between May and August 2010, content relevant to the search topic (either the whole page or, in the case of very long pages, a section of the page) from each source was extracted by two reviewers working separately. Content was then compared and a consensus reached on the content to be included in the rating assessment. The content for the rating assessments was blinded by removing any information that could identify the source sites. Word counts for the topics are given in Table 1. The order of each source was randomized using the list randomizer at www.random.org. Ethical approval was not required.
Table 1. Word counts for topics
NIMH, National Institute of Mental Health; NHS, National Health Service; ECT, electroconvulsive therapy; ADHD, attention deficit hyperactivity disorder.
Participants
An evaluation group was formed comprising three psychologists with clinical and research expertise in depression and three in schizophrenia. These experts rated each of the topics related to depression or schizophrenia respectively.
Source assessment
The content of each website was rated on a five-point scale in the following domains: accuracy, up-to-dateness, breadth of coverage, referencing and readability. The following explicit anchors for points 1, 3 and 5 were used:
-
• Accuracy: 1=many errors of fact or unsubstantiated opinions, 3=some errors of fact or unsubstantiated opinions, 5=all information factually accurate
-
• Up-to-dateness: 1=generally not up-to-date, 3=information partly up-to-date, 5=all information up-to-date
-
• Breadth of coverage: 1=limited or no coverage of topics, 3=several topics covered, 5=a broad range of topics covered
-
• Referencing: 1=no referencing, 3=partial referencing of statements or referencing with secondary sources, 5=statements are consistently referenced
-
• Readability: 1=readability suitable for someone with university education, 3=readability suitable for someone who has completed secondary education, 5=readability suitable for someone with some secondary education
Initially, raters met as a group to discuss rating procedures. A ‘pilot’ rating exercise was then undertaken with a subsequent meeting of the group to discuss and resolve differences of opinion. After the rating was completed and agreement between the raters assessed (see below), domains for which the mean intraclass correlation coefficients (ICCs) fell below 0.5 were noted. For these domains, raters were asked to meet and come to a consensus on the ratings. Agreement was re-evaluated after this process.
Readability was also assessed using the Flesch–Kincaid Grade Level Index, an objective measure of the level of reading difficulty of text, which is scaled to reflect the number of years of education required to read the text. The index reflects sentence length and word complexity (number of syllables) (Kincaid et al. Reference Kincaid, Fishburne, Rogers and Chissom1975). The index was calculated for each topic from each source using the Readability Calculator at www.online-utility.org.
Statistical analysis
Agreement between the three raters was assessed for ratings of each topic in each domain. ICCs were calculated for the average of the ratings using a mixed effects model (McGraw & Wong, Reference McGraw and Wong1996). Differences between resources within each domain were investigated using mixed-models ANOVA with resource and topic as fixed factors and rater as a random factor. To assist interpretation, a pseudo R 2 value was calculated for each factor in the model. This index was calculated as the residual variance reduction that resulted from adding each term to the model as a proportion of the residual variance of the model including only an intercept term (Singer & Willett, Reference Singer and Willett2003). Unlike the F tests reported, this index may be sensitive to the order in which terms are added to the model. However, reversing the introduction of ‘Information Source’ and ‘Topic’ made almost no difference to the results.
Because our interest was in the quality of an information source as a whole, rather than individual topics in a source, interpretation focused on the main effect of source. Sources were ordered by average ratings across all domains and topics.
Results
Inter-rater reliability
Mean, minimum and maximum ICCs for average ratings for the schizophrenia topics in each domain were as follows: accuracy: 0.82 (0.76–0.89); breadth: 0.59 (0.25–0.87); up-to-dateness: 0.83 (0.73–0.92); referencing: 0.84 (0.72–0.95); and readability: 0.69 (0.60–0.78). For the schizophrenia topics, with the exception of ratings of breadth, agreement was high and statistically significant for all ratings. Agreement regarding breadth of coverage for side-effects of antipsychotics and for cannabis and psychosis/schizophrenia were notably lower than for other topics (ICC=0.43, p=0.106 and ICC=0.25, p=0.205, respectively).
Mean, minimum and maximum ICCs for average ratings for the depression topics in each domain were as follows: accuracy: 0.59 (0.20–0.82); breadth: 0.88 (0.74–0.96); up-to-dateness: 0.75 (0.60–0.91); referencing: 0.89 (0.83–0.94); and readability: 0.87 (0.83–0.90). For the depression topics, with the exception of ratings of accuracy, agreement was high and statistically significant for all ratings. The low ICC regarding the accuracy of material on the topic of gambling and depression (ICC=0.20, p=0.254) was due to low between-resource variation rather than poor absolute agreement, as reviewers agreed completely for most resources and differed by no more than a point for others.
Expert quality ratings
In all domains, quality varied significantly between sources according to topic, but the strongest effects were between sources (Tables 2 and 3). In general, greater differences between ratings of resources were observed for schizophrenia than for depression, with notable diversity of ratings for individual topics in particular sources.
Table 2. Mixed-model ANOVA of ratings of schizophrenia information for five domains by resource and topic
a F tests have 11 112; 4 112; and 41 112 degrees of freedom, respectively.
Table 3. Mixed-model ANOVA of ratings of depression information for five domains by resource and topic
a F tests have 1193, 1194; 493, 494; and 3393, 3394 degrees of freedom, respectively.
Schizophrenia ratings
Figure 1 depicts average ratings for each resource in each domain for schizophrenia (along with the minimum and maximum average ratings for each topic in each domain). Wikipedia received the highest ratings for accuracy and was rated consistently for all topics. Accuracy was rated as being at least ‘average’ for all resources on most topics. Most resources were rated around the average level on breadth of coverage, with the Kaplan & Sadock textbook receiving the highest ratings. Some resources showed substantial variability in ratings of breadth across different topics.
Fig. 1. Average rating for 11 internet resources and a psychiatry text on five domains for schizophrenia. Bars show the minimum and maximum rating for individual topics.
For up-to-dateness, most sources were rated in the average to good range, with two (Mentalhealth.com and Encyclopaedia Britannica) being rated as poorer. Kaplan & Sadock was rated among the best sources in this regard, and consistently so across topics. Very few sources were well rated on referencing, although ratings for some topics were ‘average’. Wikipedia was clearly the most highly rated on this domain. Readability for many sources was rated as above average. Wikipedia and Kaplan & Sadock received the poorest ratings.
Depression ratings
Figure 2 depicts average ratings for each source in each domain for depression (along with the minimum and maximum average ratings for each topic in each domain). For accuracy, compared to other domains rated, there was comparatively little variation between sources and topics. As with the schizophrenia topics, Wikipedia was rated highest on average, although this website had comparatively large variation across different topics. Wikipedia, National Institute of Mental Health (nimh.nih.gov), webmd.com and Kaplan & Sadock were rated as having above average breadth of coverage within the topics studied, whereas depression.com and National Health Service (nhs.uk) had poor coverage. Other resources fell in the intermediate band.
Fig. 2. Average rating for 11 internet resources and a psychiatry text on five domains for depression. Bars show the minimum and maximum rating for individual topics.
Wikipedia was clearly the most highly rated resource on the domain of up-to-dateness. Resources varied substantially in their level of referencing, with many providing few or no citations of the medical literature. Ratings were relatively consistent across topics. It is notable that several online resources, notably Wikipedia and NIMH, were rated as comparable to, or better than, Kaplan & Sadock. Rated readability varied widely between resources and, to some extent, negatively mirrored referencing, in that resources with fewer references were rated as being more readable and vice versa. There were exceptions to this pattern, with Kaplan & Sadock being rated as the least readable resource. Of the online resources, Wikipedia was rated the least readable, although some of its topics received an average rating.
Flesch–Kincaid Grade Level Indices
Figures 3 and 4 show the Flesch–Kincaid Grade Level averaged over topics for each information source. For depression sources, the textbook was evaluated as requiring tertiary levels of education to read. This is perhaps not surprising given the intended audience of the book. However, five other sources were evaluated as requiring higher levels of education than completion of secondary schooling to be read effectively. Among these was Wikipedia. The reading level for Encyclopaedia Britannica was comparably high. Only three sources had average levels clearly less than high school completion.
Fig. 3. Flesch–Kincaid Grade Level indices for schizophrenia resources averaged over topics. Bars indicate highest and lowest levels for individual topics within a resource.
Fig. 4. Flesch–Kincaid Grade Level indices for depression resources averaged over topics. Bars indicate highest and lowest levels for individual topics within a resource.
The results for schizophrenia sources were similar to depression, with the textbook, Wikipedia and Encyclopaedia Britannica having high scores. Average reading levels were slightly higher than for depression, with only two resources, WebMD and Mentalhealth.com, having an average level of 12 years of education (equivalent to high school completion) or below.
Discussion
The quality of information about depression and schizophrenia on Wikipedia was generally rated higher than other centrally controlled resources, including 14 mental health-related websites, Encyclopaedia Britannica and Kaplan & Sadock's Comprehensive Textbook of Psychiatry. These findings may help to answer one of the most commonly raised concerns about collaboratively created websites, namely how ‘good’ is the information found there? In the case of information about topics relating to depression and schizophrenia, particularly those that are relatively controversial and rapidly evolving, the answer seems to be that the quality is relatively high, as rated by experts in the field. These findings largely parallel those of other recent studies of the quality of health information on Wikipedia, including those that have assessed the quality of information on drugs (Clauson et al. Reference Clauson, Polen, Boulos and Dzenowagis2008), on surgical procedures (Devgan et al. Reference Devgan, Powe, Blakey and Makary2007), for medical students (Pender et al. Reference Pender, Lasserre, Del Mar, Kruesi and Anuradha2009), nursing students (Haigh, Reference Haigh2010), for use in a laboratory observations database (Friedlin & McDonald, Reference Friedlin and McDonald2010), on gastroenterological conditions (Czarnecka-Kujawa et al. Reference Czarnecka-Kujawa, Abdalian and Grover2008), cancer (Rajagopalan et al. Reference Rajagopalan, Khanna, Stott, Leiter, Showalter and Dicker2010) and pathology informatics (Kim et al. Reference Kim, Gudewicz, Dighe and Gilbertson2010). Despite variability in the methodologies and conclusions of these studies, the overall implication is that Wikipedia articles on health topics typically contain relatively few factual errors, although they may lack breadth of coverage. They are also generally well referenced, but not always easy to understand (Heilman et al. Reference Heilman, Kemmann, Bonert, Chatterjee, Ragar, Beards, Iberri, Harvey, Thomas, Stomp, Martone, Lodge, Vondracek, de Wolff, Liber, Grover, Vickers, Mesko and Laurent2011).
In rapidly evolving fields such as health, a potential strength of web-based information is the ease of updating information offered by this platform. This has led to the claim that traditional peer-reviewed medical articles may be made obsolete by the advent of Wikipedia (Frishauf, Reference Frishauf2006). As might be expected, Wikipedia was the most highly rated source on the domain of up-to-dateness. However, it is noteworthy that most online sources did not eclipse the rating achieved by the Kaplan & Sadock textbook (which is typically updated every 4–5 years), although there was considerable variability across topics. This suggests that many centrally controlled websites do not exploit opportunities to update information, or they may not have the required resources to do so. Consistent with this conclusion, a recent trial found that assessment of the quality of website information and feedback to web administrators did not lead to improvement (Jorm et al. Reference Jorm, Fischer and Oh2010).
There are several limitations to this study, including the extent to which some of the ratings are subjective and may be subject to bias, particularly as the raters were working at the same institution. However, this limitation may be considered in the broader context of the issue of expert rating of the quality of scientific information, including that of peer review, which, while widely used, is generally considered to have limited evidence of validity (Jefferson et al. Reference Jefferson, Wager and Davidoff2002). In addition, the large variability of coverage between topics, which was a feature of the better-rated resources, may limit conclusions regarding overall site quality. Furthermore, care must be exercised in interpreting the absolute values of the Flesch–Kincaid Grade Level indices as it was developed and has been evaluated in a different context to medical communication. The topics covered require use of long, multisyllabic words to which the index is sensitive. However, it is clear that most of the resources make reading demands that would exceed the capacity of many users. None had reading levels consistent with primary completion/early secondary school level, despite approximately half of those in many developed countries having a reading age equivalent to primary school completion (Office for National Statistics, 1996; National Work Group on Literacy and Health, 1998) Few, if any, would meet criteria for formal patient information material or plain language statements for trial participant recruitment (Paasche-Orlow et al. Reference Paasche-Orlow, Taylor and Brancati2003). Further research should aim to discover how such information affects consumer health behaviours such as help seeking and use of evidence-based treatments. Such research might involve naturalistic reports of user behaviour (Sillence et al. Reference Sillence, Briggs, Harris and Fishwick2007; Frost et al. Reference Frost, Massagli, Wicks and Heywood2008) and may be assisted by the web's move towards greater interactivity, information sharing and collaboration. A further limitation involves the comparison of the 2009 version of the Kaplan & Sadock textbook (which is unlikely to contain references to anything published after 2008 at the latest) with websites examined in 2010, which could contain later references. However, there is some evidence that, although websites containing health information have the potential to be continually updated with new information, they are in fact relatively unlikely to change over time periods of 1 or 2 years (Jorm et al. Reference Jorm, Fischer and Oh2010; Coquard et al. Reference Coquard, Fernandez, Zullino and Khazaal2011).
Despite these limitations, it seems that the participatory model of web usage and information dissemination, as exemplified by Wikipedia, does generate high-quality information about mental disorders such as depression and schizophrenia. Given the number of patients, would-be patients and concerned others using the internet to search for information on health issues, it seems that Wikipedia is an appropriate recommendation as an information source. The value of participatory sites could be further enhanced by active contributions by psychologists and members of the medical professions. Some professional organizations, such as the Association for Psychological Science, are now urging their members to contribute to wikis to improve content (Banaji, Reference Banaji2011) and it may even be argued that these professional associations should create task forces to add official statements to Wikipedia entries relevant to the field.
Acknowledgements
The study was funded through a National Health and Medical Research Council Australia Fellowship awarded to Professor A. F. Jorm. We thank A. Ross and F. Blee for their assistance.
Declaration of Interest
None.
