Introduction
Over the last 20 years, the lack of accountability within public health services worldwide has led to their adoption of performance measurement (PM) practices (Weinberg, Reference Weinberg2001). Over this time, PM has principally been used to ensure that providers are held responsible for the quality of care provided (Clarkson & Challis, Reference Clarkson and Challis2002). However, equally significant is the role PM plays in driving service improvements and aiding performance management (Baars etal. Reference Baars, Evers, Arntz and van Merode2009). Accurate measurement of performance enables services to identify where improvements are needed and how to act on them. In addition, the information gathered informs performance management, that is, organisational planning and decision making. For example, objective caseloads can be determined on the basis of previous PM data (see Table 1 for a description of the key points of PM).
Table 1 The key points of performance measurement
PM, Performance measurement; PI, performance indicators.
Baars etal. (Reference Baars, Evers, Arntz and van Merode2009).
PM has been a central component of Irish health service policy since the mid-90s. However, PM in mental health services did not enter the spotlight until 2006, with the publication of A Vision for Change [VFC; Department of Health (DoH), 2006]. VFC called for improved PM procedures and a national minimum data set (NMDS) of performance indicators (PIs) for mental health services. However, according to a recent review conducted by Indecon (2010), these policy developments are yet to be fully implemented.
Historically, mental health services have been neglected both in terms of PIs and resource allocation (Glover, Reference Glover1995; Indecon, 2010). In comparison with medical-based hospital services, a comprehensive set of PIs have yet to be fully implemented for mental health services (Adair etal. Reference Adair, Simpson, Birdsell, Omelchuk, Casebeer, Gardiner, Newman, Beckie, Clelland, Hayden and Beausejour2003). The current lack of a ‘comprehensive’ suite of PIs specific to mental health services renders these services largely invisible to senior health service managers. This is of significant importance, given the current economic recession, in which the Health Service Executive (HSE) cut €53 million from its mental health and primary care budget (Wall, Reference Wall2012). Developing a comprehensive suite of PIs for mental health services will help managers to gather the necessary data required to compete for these limited resources.
Currently, the PIs that are measured tend to reflect the objectives of the HSE and the DoH. This is associated with a traditional top-down approach to PM (Merrigan, Reference Merrigan2007). As a result of this approach, national PM objectives are not aligned with the objectives of individual services (Clarkson & Challis, Reference Clarkson and Challis2002; Kilbourne etal. Reference Kilbourne, Keyser and Pincus2010). For example, national governments predominantly emphasise the importance of accountability. In comparison, clinicians and managers may prioritise addressing lengthy waiting lists and the need for resources. Subsequently, the importance of some national PIs is not apparent to the services measuring them. Further integration between national-level and local-level PM objectives is required.
It is hoped that this article will assist mental health service managers to conduct PM within their services. It proposes a comprehensive list of PIs that can be used at both a national level and a local service level. Managers can consult this list when determining which PIs to measure. A guide to PM at a service level is also provided. This guide focuses on a bottom-up approach to PM that illustrates how the concerns of clinicians and service managers can be integrated with the priorities of the DoH and the HSE. Using this guide will ensure accountability, drive service improvement and aid performance management at a local service level, as well as at a national level. In addition, local services will have data required to compete for limited resources.
Method
A narrative review was conducted to identify policy documents and articles relevant to PM in mental health services and/or the Irish health service. Three databases were used in this search: PsycINFO, PsycARTICLES and Pubmed. The following keywords were used: ‘performance measurement’, ‘performance management’, ‘performance indicator’, ‘mental health minimum data set’ in combination with ‘Irish mental health’, ‘Irish health service’ and with the general terms of ‘mental health’ and ‘health service’. Boolean operators (OR, AND) were used. Some articles were also identified by examining the reference lists of articles acquired in the search. Other articles were found through manual search.
Using this search method, a large body of articles was identified, relating to PM in health services, PM in mental health services, mental health service PIs, medical-based PIs, quality assurance in health services and accountability in health services. Articles that were not relevant to PM in mental health services and/or the Irish health service were disregarded. With the exception of theoretically important articles, those that were more than 10 years old were ordinarily excluded to ensure findings, and recommendations drawn from the review were up-to-date.
PI frameworks for mental health services
Donabedian's (Reference Donabedian1980) Quality of Care Model has been the primary guide for PM in health services for the last 30 years (1980). Accurate and complete PM is achieved through the consideration of PIs across three categories: structure (i.e. aspects of the service setting), process (i.e. the interactions between service users and the service) and outcome (i.e. the end results of care). According to this model, the three categories are interconnected, whereby structural factors affect the process of care that in turn influences the outcome of care.
There are also a number of useful PM tools available. A similar tool is the balanced scorecard that was originally developed for business organisations but was adapted for health services (Merode, Reference van Merode, Groothuis and Goldschmidt1999). Other tools use benchmarking systems, which tend to focus on ensuring accountability and providing financial incentives (Kilbourne etal. Reference Kilbourne, Keyser and Pincus2010). For example, in both the United States and United Kingdom, league tables are used to monitor services’ performance against agreed standards. Pay-for-performance incentives are also used whereby those services that meet specific targets are given financial rewards. In this regard, a key advantage of Donabedian's model is that it is designed to collect data that can be used to ensure accountability, as well as improve services and aid performance management.
Baars etal. (Reference Baars, Evers, Arntz and van Merode2009) incorporated this model of PM into a useful framework for mental health services, which considers a number of domains in each of the three categories (see Table 2). It is also notable that international bodies such as the World Health Organisation (WHO, 2003) and the Organisation for Economic Co-Operation and Development (Hermann & Mattke, Reference Hermann and Mattke2004) recommend using PI frameworks that are comparable with Donabedian's model and Baar's PI framework. Specific PI frameworks for national mental health services have also been developed by the United Kingdom and the United States and to a greater extent by Canada and Australia [Canadian Institute for Health Information and Statistics (CIHIS), 2000; National Mental Health Working Group (NMHWG) Information Strategy Committee Performance Indicator Drafting Group, 2005; DoH, 2009; Ohio Department of Mental Health, 2013]. An overview of these frameworks is also provided in Table 2.
Table 2 Profile of PI frameworks’ categories and domains
PM, Performance measurement; PI, performance indicators; OECD, Organisation for Economic Co-Operation and Development.
According to Indecon's (2010) review, in comparison with those used internationally, PI frameworks for mental health services in Ireland remain underdeveloped. For example, the Mental Health Commission (MHC, 2007) agreed on the development of NMDS of PIs addressing needs, inputs, processes and outcomes. Indecon has indicated that this has yet to be finalised. Furthermore, the Indecon review proposed a number of additional PIs in each of the following categories: financial inputs, facilities, human resource inputs, scope and quality of service provision, and outcomes (see Table 2). Thus, so far, this has not been fully incorporated into the HSE's PIs.
However, Ireland is not alone. PM in mental health services is at a relatively early developmental stage worldwide, which may reflect a number of wide-ranging obstacles to PM in mental health services. For example, there is a shortage of empirically supported, well-defined performance measures, from which national health services can select (Rosenheck & Cicchetti, Reference Rosenheck and Cicchetti1998). The situation is exacerbated by insufficient information and communication technology (ICT) systems, whereby data collection is often reliant on paper-based manual collection (Indecon, 2010). Furthermore, care is provided across a large number of different health-care settings (i.e. criminal justice, social services, education services, medical services, etc.) and different programme types (i.e. inpatient services, outpatient services, community services, etc.). As a result, it is difficult to apply a cohesive PM strategy for mental health services (Clarkson & Challis, Reference Clarkson and Challis2002).
Nonetheless, a national set of health service PIs are outlined in the HSE's (2013) National Service Plan. These PIs provide useful information regarding financial accountability and the volume, scope and quality of service delivery. These are monitored through a range of scorecard metrics. For example, a new web-based system called CompStat (which has recently replaced Healthstat) has been introduced to provide monthly reports of local service's performance data. For mental health services, this system is restricted to measuring the rate of admission to adult acute inpatient beds, first admission rates to adult acute units and waiting times for first appointment in child and adolescent mental health services.
A proposed suite of PIs for mental health services
Although the usefulness of currently used PIs for mental health service should not be devalued, there is a need for a comprehensive PI framework that can be applied across mental health services (i.e. inpatient, outpatient, community services, etc.). It is hoped that the PI framework proposed here will address this need. It draws from those developed by Baars etal. (Reference Baars, Evers, Arntz and van Merode2009), Indecon (2010), as well as those used internationally (CIHIS, 2000; WHO, 2003; Hermann & Mattke, Reference Hermann and Mattke2004; National Mental Health Working Group (NMHWG) Information Strategy Committee Performance Indicator Drafting Group, 2005).
Structure
Structure refers to all aspects of the service setting. It includes the resources, such as financial funding, infrastructure and personnel, available to provide high-quality care to the service's target population, as well as the characteristics of that population (Donabedian, Reference Donabedian1980). Structural characteristics such as these do not reveal whether or not a good quality of care was delivered (Berwick, Reference Berwick2003). Rather they provide an indication of the service's capacity to offer a good-quality service by diminishing or enhancing the process of care. A service has a high capacity to provide good-quality care when resources are aligned with the needs of the target population. To assess the latter in each catchment area, it is important that each service measure their allocation of finances, staff and infrastructure.
In addition, a key role of PM is to identify the population characteristics in each catchment area and highlight local patterns of deprivation (Baars etal. Reference Baars, Evers, Arntz and van Merode2009). The needs of target populations can be determined by measuring personal demographic data such as diagnosis, age, faith, gender, ethnicity, employment status, sexuality, co-morbid physical disability and co-morbid mental health disorders. These data ideally are collected early in the care pathway, at initial assessment.
Environmental characteristics are another structural element to be considered (Baars etal. Reference Baars, Evers, Arntz and van Merode2009; Byrne & Onyett, Reference Byrne and Onyett2010; National Council for the Professional Development of Nursing and Midwifery, 2010). For example, consider the organisational structure of the service. This refers to the care programme type: interdisciplinary team dynamics, interagency dynamics, the system of governance and the availability of ICT systems for data collection. In addition, Baars etal. (Reference Baars, Evers, Arntz and van Merode2009) recommend consideration of the wider regulatory, policy and professional context, by reflecting on the service's adherence to national legislation and quality standards, for example, the Mental Health Act (Government of Ireland, 2001); DoH and HSE policy, for example, VFC (DoH, 2006); and professional principles/concepts, for example, the Code of Professional Ethics of the PSI (Psychological Society of Ireland, 2008).
In general, the responsibility for the measurement of structural characteristics tends to fall to the HSE and the DoH. Although measuring structural characteristics at a service level is complex, by measuring their own, each service can ensure that the resource allocations they receive are aligned with the needs of their target population (Baars etal. Reference Baars, Evers, Arntz and van Merode2009). Perhaps what is most important is that they can highlight when this is not the case. Drawing from the studies reviewed above, a suite of PIs for measuring structure under the domains of population characteristics, resources and environmental characteristics is proposed in Table 3.
Table 3 A proposed set of PIs for measuring structure in mental health services
ICT, information and communication technology; DoH, Department of Health; HSE, Health Service Executive.
Process
Process encompasses all of the interactions service users have with the service and its service providers (Donabedian, Reference Donabedian1980). The quality of service provided during these interactions is reflected by the following: assessment numbers, first admission rates, waiting list statistics, the number of general practitioner referrals, the number of patients being treated, the number of treatment for dropouts, the length of treatment, the treatment type and communication between providers. Further insights can be gained by monitoring, for example, the percentage of service users with a particular diagnosis and/or those with a recovery-based care plan, the number offered psychological therapy and the number of involuntary committals.
Process indicators such as these can be more easily measured than some characteristics of structure or outcome. According to Mant (Reference Mant2001), process measures can provide a direct indication of the quality of care provided, whereas it is difficult to interpret the quality of care provided on the basis of outcomes exclusively. Conventionally, health service managers have used process indicators, such as lengthy waiting lists, as evidence of clinical need, in an effort to leverage additional service inputs. Nevertheless, lengthy waiting lists may merely indicate deficiencies in how services utilise inputs. Instead, process indicators may be best viewed as a useful indication of the quality of care provided. A proposed set of PIs for measuring process (drawn from the studies reviewed) is provided in Table 4. They are divided into the domains of accessibility, appropriateness, acceptability and co-ordination of care.
Table 4 A proposed set of PIs for measuring process in mental health services
PI, performance indicators.
Outcome
Outcome provides an indication of whether or not care has produced improvements in service users (Donabedian, Reference Donabedian1980). In this respect, outcomes are often considered the ‘end result’ of care and as such are deemed to be the most important aspect of care to measure. There are four essential mental health outcome domains: clinical status, functional status, quality of life and satisfaction (Indecon, 2010). A proposed set of PIs for measuring outcome in mental health services is outlined in Table 5.
Table 5 A proposed set of PIs for measuring outcome in mental health services
CORE-OM, Clinical Outcomes in Routine Evaluation-Outcome Measures.
It should be noted that having a balance of structure, process and outcome PIs is recommended to guarantee a comprehensive assessment of performance. However, for a number of reasons, fewer outcome PIs have been proposed compared with the other two categories. First, there are a limited number of evidence-based outcome measures available for mental health services (Kilbourne etal. Reference Kilbourne, Keyser and Pincus2010). Second, measures can be affected by a number of variables, such as data quality, service user case mix and other external non-therapeutic variables (Lilford etal. Reference Lilford, Mohammed, Spiegelhalter and Thomson2004). Nevertheless, the selected PIs cover the four essential mental health outcome domains and additional measures may be included for specific disorders. The evidence base for the chosen measures in Table 5 is discussed below.
The Health of the Nation Outcome Scales (Wing etal. Reference Wing, Beevor, Curtis, Park, Hadden and Burns1998) addresses 12 outcome domains including behaviour, cognition, physical health, mental health problems, social relations, general functioning, housing and other activities. There is also the Clinical Outcomes in Routine Evaluation-Outcome Measures (CORE-OM) that was developed specifically to measure outcomes of psychotherapy and counselling (CORE, 2012).
In the United Kingdom, the ethos of PM has been further strengthened by the Improving Access to Psychological Therapies programme (Reference O'Shea and ByrneO'Shea & Byrne, in press) that includes low- and high-intensity therapy for a range of mental health disorders. Throughout the therapy, sessional outcome measures are regularly completed. The information gathered provides an indication of treatment progression and aids clinical supervision. Perhaps most importantly, in terms of PM, the information also enables services to demonstrate their effectiveness.
The United Kingdom has also invested in procedures for assessing service user satisfaction with mental health services (Health Care Commission, 2008). Surveys are used to explore whether service users are being respected, being adequately listened to and whether their views regarding treatment are taken into account. Similarly, questions are investigated annually in the Irish health service through the National Service-User Executive Survey (Indecon, 2010). In 2011, the MHC and the Irish Society for Quality and Safety in Healthcare piloted an in-depth national survey in 28 inpatient services, which also assessed the provision of care plans, service users’ involvement in care plans and the adequacy of information provided to service users (Irish Society for Quality and Safety in Healthcare and the Mental Health Commission, 2012). The long-term plan for this survey was that it be used on a national level across inpatient and community services. However, this has not yet been implemented.
In terms of outcome, additional treatment factors considered by the HSE and DoH are cost-benefit (i.e. assessment of the total monetary costs and gains of treatment), cost-effectiveness (i.e. assessment of the long-term health gains achieved by providing intervention) and cost-minimisation (i.e. assessment of the difference in costs between two treatments) (Robinson, Reference Robinson1993; Hoch & Smith, Reference Hoch and Smith2006; Reference Twomey, Byrne and McHughTwomey etal. submitted). However, it is difficult to determine the financial value of some aspects of outcome, such as quality of life. Having said this, findings indicate that, given the offset costs of not providing services (e.g. unemployment and disability benefits, medical expenses, sick leave, etc.), psychological interventions can be considered ‘self-financing’ (Chisholm, Reference Chisholm1998).
A bottom-up approach to PM
Establishment of a comprehensive suite of PIs for mental health services will pave the way for PM. The suite of proposed PIs outlined above is useful both in the national and the local service context. Previously, PM was primarily considered within the national context and, as a result, PIs tended to reflect the objectives of the HSE and DoH. Sole use of this top-down approach is the key contributory factor associated with ineffective implementation of PM policy in health-care services internationally (Merrigan, Reference Merrigan2007).
A primary criticism of the top-down approach is that national objectives and targets are not fully aligned with those of individual services (Clarkson & Challis, Reference Clarkson and Challis2002; Kilbourne etal. Reference Kilbourne, Keyser and Pincus2010). Consequently, the importance of some national PIs is not apparent to the services measuring them. Further integration between national-level PIs and local service-level PIs would enable PM to be completed at multiple levels. An integrated set of PIs would also facilitate dialogue between services and the DoH and the HSE. Co-operation between local services, the DoH and the HSE is essential.
A second criticism of this approach is that there is a tendency for PM to be conducted for the primary purpose of ensuring accountability; summative judgements of care quality are deduced from performance measure data and used to compare service performance with other services or standards of care. A sole focus on accountability is particularly objectionable to managers and frontline staff as it can foster a ‘blame culture’ (Moullin, Reference Moullin2004). As a result, there is an ‘image problem’ for PM among mental health service professionals (Harnett etal. Reference Harnett, Bowles and Coughlan2009).
For these reasons, consideration of a bottom-up approach to PM is warranted. Involving frontline staff and managers ensures integration between national and local PM objectives and promotes a positive perception of PM among staff (Ballantine etal. Reference Ballantine, Brignall and Modell1998; Grote, Reference Grote2000). For example, in the Swedish health service, shifting responsibility for PM to staff improved PM procedures (Ballantine etal. Reference Ballantine, Brignall and Modell1998).
Introducing a bottom-up approach to PM is also in the interests of staff, as it would provide an opportunity to illustrate their significance and value within their services. Smith (Reference Smith2002) highlights the benefits of professional prestige, career advancement and intrinsic satisfaction in good clinical and research performance. In the case of mental health services, it also offers an opportunity for staff to address how peripheral their services are considered. Since the time of the asylums, there has been a tendency for these services to be neglected in terms of resource allocation (Glover, Reference Glover1995). By producing mental health service performance data, these services would be in a better position to compete for scarce resources.
The bottom-up approach proposed in this article illustrates how the concerns of clinicians and service managers can be integrated with the priorities of DoH and the HSE. A practical four-step guide is provided in Table 6. It is informed by a number of instructive guidelines. For example, the WHO (2003, 2005) has produced useful guidelines that can assist managers and staff to implement national PI frameworks. Additional useful tools include the Health Information and Quality Authority's (2010) Guidance on Developing Key Performance Indicators and Minimum Data Sets to Monitor Healthcare Quality; the MHC's (2007) Quality Framework for Mental Health Services; and the Mental Health Team Development Audit Tool (Byrne & Onyett, Reference Byrne and Onyett2010; Roncalli etal. Reference Roncalli, Byrne and Onyett2013). A template checklist to be used for each PI is also provided in the appendix.
Table 6 A practical guide to service-level PM
PM, performance measurement; HSE, Health Service Executive; DoH, Department of Health; PI, performance indicators.
Step 1: needs assessment
The first task for a manager is to establish a PM team that will be responsible for determining the service's PM needs, that is, which performance data to collect. As most mental health services have limited resources available, there is no advantage in measuring a large body of PIs without having the resources to interpret and respond to the data produced (McMillen etal. Reference McMillen, Zayas and Books2008).
Hence, the structure, process and outcome PIs, outlined previously, are intended as a suite of ‘proposed’ PIs for mental health services. It is not intended as a list of ‘obligatory’ PIs. Each service will identify its own suite of PIs. The chosen PIs will ideally reflect the service's improvement and performance management needs, but also the core set of PIs set out by the HSE and DoH. The latter will promote the integration of national and local PIs. To ensure that the different stakeholders are involved in the PI selection process, a ranking system such as the Delphi process could be utilised (Hermann & Mattke, Reference Hermann and Mattke2004). This allows each panel member to rate potential PIs in terms of scientific soundness and importance. In addition, there are a number of further factors to be considered (see Table 7).
Table 7 Identifying PIs
PI, performance indicators.
Bodart & Shrestha (Reference Bodart and Shrestha2000), McEwan & Goldner (Reference McEwan and Goldner2001).
Step 2: review procedures
A review of current PM procedures is the next step. First, determine which PIs are currently being measured and how they are measured. Next, identify any barriers to PM and how they can be resolved. Common barriers to PM data collection include poor ICT systems, limited time and staff resources for data collection, and unclear assignment of data collection duties among staff. In particular, without an effective ICT system, data collection is limited to paper-based episodic data collection procedures. To address this issue, VFC suggested that the NMDS for mental health services be incorporated within the HSE's electronic database (DoH, 2006), but this has not yet materialised.
Performing a ‘walk-though’ analysis of current PM data collection procedures is often the best way to identify these barriers and devise solutions. Issues can be resolved by developing uniform procedures for data collection detailing who collects the data, when the data are collected and where the data are collected.
Step 3: application
The next step is to refine PM procedures on the basis of the findings of the walk-through analysis. For example, determine data sources and the frequency of collection, and assign data collection duties. Last, a performance data report is prepared.
Step 4: evaluation
The final step involves an evaluation of PM procedures and outcomes. First, in relation to PM procedures, review the service's PM requirements, the PIs used and the data collection processes used. Second, perform an analysis of the PM outcomes to ascertain whether targets are being achieved/partially achieved/not achieved and determine follow-up actions.
Conclusion
Despite PM in the Irish mental health service becoming a focal point in recent policy documents, this narrative review shows that this vision has not yet fully materialised. This was a selective review of primarily Irish- and UK-based studies and policy. A focus was placed on PM in the Irish mental health service. Nevertheless, the authors are confident that this review provides useful applications for other Irish health services, in particular, other health- and social-care professional teams, for example, speech and language therapists, occupational therapists and social workers.
This article guides service managers through the process of establishing a suite of PIs that are useful at both a national and local level, and deploying them within their own service. Currently, the national PIs for mental health services are incomplete and provide a narrow insight into service performance. Hence, a comprehensive suite of PIs are proposed to address this issue.
Up until now, national PIs have primarily reflected the objectives of both the HSE and the DoH. An integrated set of PIs, also reflecting the objectives of local services’ clinicians and managers, is now required. To achieve this, this paper proposes a four-step bottom-up approach to PM. Although future research is required to investigate the relative efficacy of this approach, the authors are confident that it will aid PM at a local service level, as well as a national level. As we now have the tools for PM, it is time we embraced them.
Acknowledgements
The authors thank the reviewers for their feedback and suggestions which have greatly contributed to the final quality of the article.
Declaration of Interest
None.
Appendix. Template for PM checklist
