A case for action

Two key questions are worth asking: what is information and why is it so crucial for mental health (Institute of Medicine, 2006)?

There is a subtle difference between data and information. Data are the facts from which information is derived. In this paper we focus on the data that should be processed into information, which are specifically related to inputs, processes and outputs of services delivering mental health care (including primary care and substance abuse services). To progress from data to information, the former need to be contextualised. Information is defined as a meaningful collection of facts or data that gives meaning and context to the raw data (WHO, 2000).

In their seminal book, Thornicroft & Tansella (Reference Thornicroft and Tansella1999) proposed a heuristic model (the ‘matrix model’) for using information to evaluate the relative strengths and weaknesses of mental health services and systems. This model was inspired by the Donabedian classic model that provides a framework for assessing the quality of medical care (Donabedian, Reference Donabedian1988), but the authors added the geographical and temporal dimensions to this framework. The geographical dimension refers to three geographic levels (country/system, local/program and patient/individual) while the temporal dimension refers to the Donabedian triad (inputs, processes/outputs and outcomes). Using these two dimensions, the authors construct a ‘3 × 3’ matrix to highlight critical issues for mental health services. Their model stresses the importance of having information in all the cells of this matrix and defines which information at what level is needed to improve mental health care.

Information is needed by all stakeholders, at all levels. Practitioners are (or should be) interested in assessing treatment outcomes however defined (e.g., clinical, functional or satisfaction with care); managers are interested in examining the effectiveness of the departments they lead; while decision-makers want to learn the outcomes of their policies. Other stakeholders such as users, families and advocacy non-governmental organisations (NGOs) that are keen to monitor the quality of care and its respect for human rights also have an interest in information. Finally, information is needed to verify system infrastructure and the responsible and transparent utilisation of scarce resources.

There is a consensus that rational planning aimed at achieving a well-functioning system is hardly possible in the absence of sound monitoring and evaluation. Both these tasks are information-dependent strategies (Thornicroft & Tansella, Reference Thornicroft and Tansella1999). Monitoring refers to tracking key elements of program or service performance (inputs, activities, results) on a regular basis, while evaluation refers to the systematic appraisal of the value, effectiveness and impact of programs or services. Monitoring is usually an ongoing assessment, which does not take into consideration longer term impact, while evaluation is an episodic assessment of the change in targeted results that can be attributed to program/project intervention, or the analysis of inputs and outputs to determine their contribution to outcomes. Monitoring and evaluation are complementary and, to some extent, overlapping rather than entirely distinct processes. Information collected through monitoring usually feeds into systematic evaluations that are conducted, and monitoring involves some appraisal of information that can be used to inform the development of programs and services (WHO, 2007; UNDP, 2009).

Reliable and timely health information is the foundation for effective health services management and public health action (WHO, 2011b ). Persuaded that what ‘gets measured gets done,’ WHO included ‘monitoring community mental health,’ through strengthening information systems, as one of the ten recommendations that can make a difference in mental health care (WHO, 2001).

However, the question that arises is: What does occur in the real world? The World Health Organization-Assessment Instrument for Mental Health Systems (WHO-AIMS) (WHO, 2005a , 2009) and the Atlas series (WHO, 2005b , 2011a , 2015a ; Lora & Sharan, Reference Lora and Sharan2015), designed for mental health information gathering in all member states, revealed that information attracts limited investments and that its full use is more rare than frequent. The OECD (2014) stated that the quality of care of mental health disorders ‘will continue to trail behind that of other diseases until appropriate indicators are used to measure quality, and appropriate data are collected.’

There are several reasons for this discrepancy despite the progress noted above (Ryan et al. Reference Ryan, De Silva, Terver, Ochi and Eaton2015); for instance, insufficiencies in staff training, poor information infrastructure and lack of common performance standards. We propose that an historical reason is dominant: mental health services lack a tradition of accountability. This is the case not only in low- and middle-income countries (LAMICs), but also in high-income countries (HICs). Paradoxically, everywhere, the need for information for change mounts as resources dwindle. Inadequate training is one culprit: clinicians who frequently head the services have rarely been exposed to epidemiological and mental health services research. Shackled by out-dated tradition and a deficit in training, mental health trails behind the general health system in collecting and analysing standard information (Institute of Medicine, 2006; Pincus et al. Reference Pincus, Page, Druss, Appelbaum, Gottlieb and England2007; WHO, 2009, 2015a , b ). Also, there is a structural challenge for mental health especially after deinstitutionalisation. Mental health services are delivered by a heterogeneous network of public, semi-private and private settings, both dedicated and non-dedicated. All these operate with separate information systems that are driven by different and sometimes competing interests. Collecting information from this extended network is usually hindered by inadequate information technology resources and the absence of common performance measures (Kilbourne et al. Reference Kilbourne, Keyser and Pincus2010).

Assuming the Thornicroft & Tansella point of view on the use of information for mental health system change, this paper analyses the tools that are available for collecting and summarising information at system level, highlights the various goals of information gathering, discusses implementation issues and charts the way forward.

Mental health information systems

WHO (2005c ) defined a mental health information system (MHIS) as ‘a sustainable system for collecting, processing, analysing, disseminating and using information about mental health services and the mental health needs of the population it serves.’ For WHO (2007, 2010, 2011b ), MHIS is one of the building blocks of the health system, just like financing, governance, human resources and service delivery. For WHO, an MHIS makes for slow and rational decision-making, in lieu of fast intuitions (Kahneman, Reference Kahneman2011) thus leading to more effective governance and service improvement.

The following questions need to be addressed when implementing an MHIS: What purpose is served by the collected information? Who are the providers and consumers of the information? What information should be collected?

MHIS providers

The WHO proposed a multi-level model of a mental health services organisation, using a pyramid framework (Levav, Reference Levav and Levav1992; WHO, 2003b ) (Fig. 1) that could help in defining the information providers.

Fig. 1 Multi-level model of a mental health services organization, using a pyramid framework.

Accordingly, the MHIS should be able to collect information from all formal mental health services (for adults, children and adolescents, and the elderly), from substance use health services (that often remain separated) and from mental health services delivered at the primary care level. Furthermore, in view of growing awareness of the need to approach mental health problems inter-sectorally, the MHIS should also be able to incorporate information from others areas (e.g., social services, education, criminal justice, labour, as well as from NGOs), invariably with proper ethical safeguards.

Mental health indicators

Once data has been collected, how should it be synthesised? Indicators are needed for this purpose: they summarise information relevant to the mental health services and the population they serve, measuring changes over time (WHO, 2005c ). Even if quantitative indicators are often criticised for measuring what is easy to measure while ignoring what is relevant to quality (Legido-Quigley et al. Reference Legido-Quigley, Mckee, Walshe, Sunol, Nolte and Klazinga2008), they nevertheless provide the basic data required for benchmarking and accountability.

Indicators may derive from different data sources. The more relevant source is the MHIS itself that collects data that are often converted to indicators, adding a denominator to a raw numerator (e.g., rate of outpatients per 100 000). Also, vital statistics and community surveys may enrich the set of indicators collected at country level. The use of the former is needed to evaluate two key outcome indicators: general mortality and, specifically, deaths by suicide (WHO, 2014). Mental health surveys carried out in HICs to assess the mental health of the population are excellent sources of indicators (e.g., percentage of the population with identifiable mental disorders but not receiving care) (Sunderland & Finlay, Reference Sunderland and Finlay2013).

Indicators may be related to inputs (human resources, facilities, financing, etc.), process/outputs (activities delivered by mental health services, i.e., hospitalisations, psychotherapies, medications) and outcomes (the effect of services on the mental health of the population being served in terms of mortality, symptoms reduction, improvement of social functioning and quality of life). Indicator systems routinely collect information on input and process/output but less frequently on outcome, which is more difficult to obtain and analyse (Thornicroft & Tansella, Reference Thornicroft and Tansella1999).

Each measure has its strengths and limitations (Kilbourne et al. Reference Kilbourne, Keyser and Pincus2010). For example, input measures, although relatively simple to collect and analyse, do not inform about outputs that were delivered. Another example refers to process measures that have been widely used for performance measurement. These appeal to providers because they represent activities over which they have the most control. However, the quantity of outpatient visits is not related to the quality of service delivered. Outcome measures are often requested by managers because they assess whether services are effective. But these measures are quite expensive and require case mix adjustments to ensure that observed differences in outcomes are not due to clinical differences in severity of illness across users (Hoenders et al. Reference Hoenders, Bos, Bartels-Velthuis, Vollbehr, van der Ploeg, de Jonge and de Jong2014). Yet, it is possible, from routinely collected data, to assess the reduced life expectancy of patients with severe mental disorders (Lesage et al. Reference Lesage, Rochette, Emond, Pelletier, St-Laurent, Diallo and Kisely2015); or to use the life expectancy metrics to compare the effectiveness of psychotropic medication in the real life of a population (Tiihonen et al. Reference Tiihonen, Lönnqvist, Wahlbeck, Klaukka, Niskanen, Tanskanen and Haukka2009).

In terms of domains covered, curative care is predominantly addressed by mental health indicator systems, although indicators related to rehabilitation do exist (Killaspy et al. Reference Killaspy, White, Wright, Taylor, Turton, Schützwohl, Schuster, Cervilla, Brangier, Raboch, Kališová, Onchev, Alexiev, Mezzina, Ridente, Wiersma, Visser, Kiejna, Adamowski, Ploumpidis, Gonidakis, Caldas De Almeida, Cardoso and King2011). Two other domains, often neglected, are primary prevention and promotion. For example, the Mental Health Atlas of 2014 investigated at system level that 41% of the member states have at least two functioning promotion and prevention programs in existence. But indicators on these activities are rarely collected at system level: a report of the European Commission (2013) found that only two participating countries (Denmark and the Netherlands) appeared to be collecting information on prevention or promotion activities.

Monitoring mental health care delivery

Monitoring activities delivered by mental health services is a common use of MHIS. It represents a first step in analysing the mental health system of a country and improving accountability. MHIS monitors if services are delivering the kind and amount of activities that they were expected to deliver.

The following indicators should be used in performance analysis. What is needed is the analysis of access to services, including equity, using treated prevalence and incidence rates, and service delivery in terms of rate of outpatient contacts, admissions and days spent in inpatient facilities or day care settings and the balance between in- and out-patient activities (Lund & Fisher, Reference Lund and Flisher2003). Lately, coverage as an indicator of access to mental health services has received increasing attention (De Silva et al. Reference De Silva, Lee, Fuhr, Rathod, Chisholm, Schellenberg and Patel2014). It has been one of the main indicators for monitoring the WHO 2013–2020 Action Plan, and is of crucial importance in assessing treatment gap (Kohn et al. Reference Kohn, Saxena, Levav and Saraceno2004; Lora et al. Reference Lora, Kohn, Levav, McBain, Morris and Saxena2012b ).

There are examples of HICs, e.g., the UK (HSCIC, 2014) and Canada (MHCC, 2015), where basic data on mental health services are routinely provided. Also, a large number of LAMICs, about 90 in all, reported to WHO-AIMS (WHO, 2015b ) that they monitor service provision. While an MHIS is sufficient in monitoring most activities of the mental health services, psychiatric case registers are particularly useful in monitoring more complex indicators than service delivery and have been utilised this way in some countries such as Denmark, Israel and Italy (Lombardy). An example of using case registers to analyse these indicators at system level (e.g., treated prevalence and incidence, continuity of care, patterns of care) has been described elsewhere (Lora et al. Reference Lora, Barbato, Cerati, Erlicher and Percudani2012a , Reference Lora, Kohn, Levav, McBain, Morris and Saxena b ).

The high rate of comorbidity of mental and physical health problems has led to calls for integration of physical and mental health services. Integration could be achieved in many ways, including linking clinical information systems. An example is the use of the chronic diseases surveillance system based on health administrative databases (activities of medical providers in hospitals, emergency rooms, outpatient clinics and primary care settings). Typically, they record age, gender, diagnoses, local area social deprivation, mortality and causes of deaths (PHAC, 2015). More recent developments in electronic medical records in primary care also allow their linkages with other health administrative databases (John et al. Reference John, Dennis, Kosnes, Gunnell, Scourfield, Ford and Lloyd2014 in Wales; Morkem et al. Reference Morkem, Barber, Williamson and Patten2015 in Canada).

Evaluating the quality of mental health care

To achieve this goal, clinical or quality indicators are needed. They compare the quality of care delivered against evidence-based standards of care, establish benchmarking, support accountability and promote quality improvement (Mainz, Reference Mainz2003). Clinical indicators are often derived from administrative data provided by information systems, thus avoiding the burden of ‘ad hoc’ data collection. Examples of clinical indicators include the percentage of users that received outpatient care in the 30 days following discharge from an inpatient facility (OECD, 2014; WHO, 2015a ) or, with regard to appropriateness, the percentage of users who received an adequate dosage of drug therapy at the onset of a depressive episode (CQAIHM, 2015).

In the past decade, there has been an increase in the development of quality measures (Lauriks et al. Reference Lauriks, Buster, de Wit, Arah and Klazinga2012), since evidence from quality measurement and quality monitoring, combined with feedback, auditing and public disclosure of measurement data, are useful means of improving care (Mainz & Bartels, Reference Mainz and Bartels2006).

From an international perspective, the OECD (Reference Armesto, Medeiros and Wei2008), starting from the Health Care Quality Indicators project (Hermann et al. Reference Hermann, Mattke, Somekh, Silfverhielm, Goldner, Glover, Pirkis, Mainz and Chan2006) and analysing the information available at country level, found that only few indicators were available among member states – hospital re-admissions for users with treated psychiatric disorders and mortality for persons with severe psychiatric disorders. These indicators should be considered as the ‘low hanging fruit’ immediately available to start the data collection. The OECD (2014) also recommended a longer list of indicators (Box 1) used in monitoring the quality of mental health care.

At the national level, several countries in Europe (Bramesfeld et al. Reference Bramesfeld, Amaddeo, Caldas-de-Almeida, Cordosa, Depaigne-Loth, Derenne, Donisi, Lindelius, Lora, Mainz, Mulders, Szecsenyi and Killaspy2016; Lora et al. Reference Lora, Monzani, Ibrahim, Soranna and Corrao2016), as well as Canada, Australia and New Zealand, are taking the lead to develop and implement mental health performance measures as part of their national health systems (Spaeth-Rublee et al. Reference Spaeth-Rublee, Pincus and Huynh2010). In the USA, organisations such as the NCQA (2015), the NQF (2015), SAHMSA (2015) (Substance Abuse and Mental Health Services) and the Department of Veterans Affairs (VA) are intensively working in this field.

Supporting research

The MHIS is a powerful tool for health services research. Studies could use data from the MHIS, or link bits of information through citizens’ or residents’ unique personal identification number with other data sources like pharmaceutical prescriptions, health interventions, census data, other registers and biobanks, with adequate considerations of confidentiality and privacy.

The list of possible uses of MHIS data for mental health services research is long: from sampling frames in epidemiological studies (Levine et al. Reference Levine, Lurie, Kohn and Levav2011), to studies on risk factors and treatment outcome (Levine et al. Reference Levine, Levav, Yoffe and Pugachova2014; Goldberger et al. Reference Goldberger, Haklai, Pugachova and Levav2015) and to pharmaco-epidemiology (Barbui et al. Reference Barbui, Conti, Purgato, Cipriani, Fortino, Rivolta and Lora2013). In this area, psychiatric case registers are particularly useful (Wierdsma et al. Reference Wierdsma, Sytema, van Os and Mulder2008; Munk-Jørgensen et al. Reference Munk-Jørgensen, Okkels, Golberg, Ruggeri and Thornicroft2014), although their use is expensive and limited to HICs. Joining health administrative databases in public managed care countries like the UK or Canada, allows the study of primary and specialist mental health services as determinants of outcomes just like suicide or excess mortality of people with mental disorders (John et al. Reference John, Dennis, Kosnes, Gunnell, Scourfield, Ford and Lloyd2014; Lesage et al. Reference Lesage, Rochette, Emond, Pelletier, St-Laurent, Diallo and Kisely2015).

Monitoring human rights

MHIS are essential for reducing violation of human rights in care facilities (Barbato, Reference Barbato, Lindert and Levav2015). For example, data use is a key strategy for reducing restraints and seclusion. Data collection and reporting provides staff with an effective feedback loop, allows for benchmarking and highlights the organisational commitment to change (especially when data are widely reported) (Scanlan, Reference Scanlan2010). An international initiative sought to develop a consensus framework for mental health quality measures through a Delphi process (Parameswaran et al. Reference Parameswaran, Spaeth-Rublee and Pincus2014). Indicators with regard to respect of human rights (seclusion, involuntary treatment, compulsory hospitalisation, restraint) were ranked among the ten most important indicators. As examples of good practices, some countries such as Australia (AIHW, 2014), routinely monitor restraint and seclusion.

For WHO, human rights observance is a cornerstone of quality of care. Accordingly, it stresses the use of information for this goal. In a report on 42 countries (WHO, 2009) data showed that a number of countries paid scant attention to the human rights of service users. Mental health legislation exists in only half the countries, human rights inspections and training are infrequent and data collection on involuntary admissions and physical restraint and seclusion is limited. About half of the participating countries did not provide data on involuntary admissions in mental hospitals and about one third in community-based inpatient units. Information on physical restraint and seclusion in inpatient units were collected by only half of the reporting countries. Moreover, in the 2014 Atlas (WHO, 2015a ), only 79 of 171 countries provided data about the proportion of involuntary admissions to mental hospitals, general hospital psychiatric wards and community residential facilities.

Financing the mental health system

At a system level, mental health budgets and human resources represent inputs, alongside numbers of people in need of care, mental health plans and policies. The WHO Mental Health Atlas (2015a ) contains financial and human resource indicators such as mental health expenditures per capita for mental hospitals, other inpatient and day care facilities and outpatient and primary care facilities. Despite the importance of these items, only 17 high-income, 16 middle-income and 7 low-income countries were able to provide the minimal information. Sources of information in any country will depend on differences in coverage by the public, semi-private and private managed care systems and the capacity to measure services and attribute item costs. In any country, differences in funding (hospital v. outpatient, health v. social services, drug coverage by various v. single payers) will complicate the capacity to produce a population-based assessment of overall and detailed budgets, with indicators that allow at least intra-country regional comparisons, and eventually international comparisons.

Information is used to identify the benchmarks for a balanced mental health system budget. Wing et al. (Reference Wing, Brewin, Thornicroft and Thornicroft2001) proposed a population-based approach for the needed services and costs of a balanced mental health care system for users with severe mental disorders. A similar exercise, using Canadian benchmarks for number of beds, residential facilities, intensive home care, supplement to rent; and costing of each of these services, suggested an annual per capita of 134 Canadian dollars per inhabitant (Lesage, Reference Lesage2014). It may involve an increase in the current mental health budgets which, in Canada, currently represents 5–6% of the public managed health and social services budget (Jacobs et al. Reference Jacobs, Yim, Ohinmaa, Eng, Dewa, Bland, Block and Slomp2008).

Information from different psychiatric settings is needed to create new financing models such as bundled payments for mental health services, where mental health services receive a single payment for treating a patient with a specific mental health condition across a full cycle of care, from the hospital to the community (NCMH, 2014). Recently, the USA (Bremer et al. Reference Bremer, Scholle, Keyser, Houtsinger and Pincus2008) and the UK (NHS, 2013), moved a step further and established financial incentives, such as pay-for-performance, where clinicians and organisations receive bonuses if they meet certain quality thresholds based on quality indicators.

Assessing the impact of policy

Information is still underutilised in monitoring the implementation of mental health policies and evaluating their impact. An MHIS may support studies on the impact of policy changes by linking information at the system level on inputs, processes and outcomes, information on costs, availability, access, health services delivery, quality and outcomes of care for individuals with mental disorders (WHO, 2007). In Australia, the newly created National Mental Health Commission (2012, 2016) is producing a publicly available report card on the performance of the mental health system by aggregating indicators from various sources. It provides ‘an independent eye over how we as a nation support the estimated 3.2 million Australians each year who live with a mental health difficulty, their families and how we provide and co-ordinate the services they need’.

At the international level, the WHO tried to determine whether the countries had utilised the information collected through WHO-AIMS to strengthen their mental health systems (WHO, 2009) by asking them if results had been used to develop or revise a mental health policy, plan or for another planning purpose. Almost half the countries have used WHO-AIMS to develop or revise a mental health policy or plan and an additional eight countries (19%) were either in the process of or were planning to use WHO-AIMS for this purpose. Fifty-five per cent of countries have used WHO-AIMS for some other planning purpose.