Introduction
Central to the notion of recovery is the genuine involvement of service users and their families as equal partners in the planning of their care and also in the development, delivery and evaluation of mental health services (Department of Health and Children, 2008; World Health Organization, 2010). The collaborative model contrasts with the traditional clinician-centred psychoeducational programmes that can position service users as passive recipients of information and can reinforce their powerlessness and disengagement from the mental health system (Jormfeldt et al. Reference Jormfeldt, Rask, Brunt, Bengtsson and Svedberg2012). Peer involvement is part of a process that seeks to balance scientific evidence with expertise by experience, and to reconfigure hierarchical relationships between practitioners, users of services and family members (Mental Health Commission, 2005; Higgins et al. Reference Higgins, Callaghan, deVries, Keogh, Morrissey, Nash, Ryan, Gijels and Carter2012). Recognition is given to the value of experiential knowledge and expertise by experience as well as the unique support, empathy and inspiration derived from peers (Solomon et al. Reference Solomon, Draine, Mannion and Meisel1997; Watts, Reference Watts2014).
Issues of empowerment and self-advocacy are core to achieving equal involvement of service users and carers within the mental health services (McDaid, Reference McDaid2006). Given the dearth of information and lack of communication experienced by service users and their families within Irish mental health services (Western Health Board and Schizophrenia Ireland, 2002; Brosnan, Reference Compton, Quintero and Esterberg2006; Crowe, 2006; Dunne, Reference Dunne2006; Kartalova-O’Doherty et al. Reference Kartalova-O’Doherty, Doherty and Walsh2008; National Service User Executive, 2011) and their unequivocal right to access information (Mental Health Commission, 2005; Department of Health and Children, Reference Borkin, Steffan, Krzton, Wishnick, Wilder and Yangarber2006; Higgins, Reference Higgins2008; World Health Organization, 2010), information programmes are an important and powerful mechanism by which service users and their families can be more empowered to become involved in the mental health system (Mental Health Commission, 2005; McDaid, Reference McDaid2006).
Given the potential benefits derived from both peer and clinician involvement in the delivery of education, the EOLAS programmes were developed to address information gaps. As described in the previous paper, the EOLAS programmes are parallel clinician and peer-led information programmes for people with enduring mental health problems (specifically schizophrenia spectrum and bipolar disorders), and for their family members and significant others (Higgins et al. Reference Higgins, Hevey, Gibbons, O’Connor, Boyd and McBennett2015). Both programmes have been co-produced, co-delivered and co-managed by users (service users and family members) in conjunction with mental health practitioners. This paper presents an evaluation of the pilot EOLAS programmes.
Objectives of evaluation
The objectives of the evaluation were:
-
∙ to investigate the impact of the information programmes on service users’ and family members’ perceived knowledge, attitudes towards recovery, hope, support, advocacy and well-being;
-
∙ to explore participants’ experience of programme involvement, from the perspective of participants and facilitators;
-
∙ to determine ways of improving and developing the EOLAS programmes.
Research design
The evaluation used a sequential mixed method design involving the collection of both quantitative and qualitative data. A pre-post design was used to collect quantitative data using questionnaires completed before and after programme participation. The questionnaires gathered data to provide preliminary data on the impact of EOLAS on salient outcomes. In addition, semi-structured interviews with participants, facilitators and project workers explored their experiences and views of the programme to provide a more in-depth understanding of the impact of the EOLAS programme on the lived experience of those who delivered and received the intervention. The mixed methods addressed different research questions; the focus on the interviews was determined in advance of the study and was not guided by the results of the quantitative analyses.
The pre-programme service user questionnaire collected demographic data, including information on age, gender and relationship with the person with the mental health issue. Section 2 consisted of a 15-item scale assessing participants’ knowledge of mental health issues including symptoms, coping skills, relapse triggers and roles within the community mental health team. The scale was developed by the research team to ensure that the items reflected the content of the programme; where appropriate the items were drawn from similar knowledge scales (e.g. Compton et al. Reference Crowe2007) but items also had to be generated to relect multidisciplinary team (MDT) roles within Irish healthcare service systems. The scale was internally consistent with an α of 0.88. Subsequent sections comprised of validated scales including the Recovery Attitudes Questionnaire 7 (RAQ-7), which consisted of a seven-item scale developed by Borkin et al. (Reference Brashers, Haas and Neidig2000); the Herth Hope Index, which measured respondents’ level of hopefulness for the future (Herth, Reference Herth1992); a 10-item self-advocacy scale taken from the 12-item Brashers et al.’s (Reference Brosnan1999) Patient Self-Advocacy Scale. Drug attitudes were measured by five items taken from the 10-item Drug Attitude Inventory (DAI-10) (Hogan et al. Reference Hogan, Awad and Eastwood1983). The pre-programme family member questionnaire replicated the service user survey except there were additional sections including the General Health Questionnaire (GHQ-12; Goldberg et al. Reference Goldberg, Gater, Sartorius, Ustun, Piccinelli, Gureje and Rutter1997) and the Social Network of Support Scale. The Social Network of Support Scale was devised by the EOLAS research sub-committee and included six items examining the family members’/friends’ perceptions of the social support that is available to them to help them cope with their caregiving responsibilities, as well as to deal with their own needs. The scale items and response options were based on existing measures of carer social support (e.g. Pearlin et al. Reference Pearlin, Mullan, Semple and Skaff1990) but the content was tailored to cover the aspects of support that EOLAS addresses. The scale was internally consistent with an α of 0.88.
The surveys were piloted with a group of users and family members. The feedback from the pilot indicated that the questionnaires were user-friendly and could be completed within the suggested timeslot of 20 minutes. The post-programme questionnaires repeated the questions included in the pre-programme questionnaire. An additional section examined participants’ satisfaction with the programmes and their perceptions of the operation of the programmes, of the learning tools and teaching methods. Participants were also asked for any other comments they would like to make about the EOLAS project.
Post-programme semi-structured interviews further detailed the participants’ views on the programmes. Individual interviews were held with participants 2–3 weeks post-programme, and were guided by a topic guide, which prompted participants to discuss their personal experience of the programmes, the impact on their lives, comparisons with other mental health programmes and recommendations for further programme development (see Table 1).
Table 1 Semi-structured interview topic guide
Recruitment
During the recruitment process, potential participants were informed by clinical staff that the programme was being evaluated by a group that were not part of the clinical team and that they could participate in the EOLAS programmes without obligation to participate in the evaluation. Participants were also given an information brochure with details of the study including its purpose and process, and assuring confidentiality. Potential participants had at least a week to read the information before the start of the programme. On the first day of the programme, participants were given a pre-programme questionnaire in an envelope along with a research information sheet. They were requested to complete the questionnaire if they wished to be involved and return it in the envelope provided. Participants were requested not to write their names or any other personal information on the questionnaires. Participants who did not wish to complete the questionnaire were advised that they could return the questionnaire without completing it. Consequently, people who did not wish to participate could do so anonymously. The post-programme questionnaires were distributed by the researchers at the end of the final EOLAS session. The post-programme pack included an opt-in form for the semi-structured interviews. Participants could complete the questionnaire immediately or take it away and return to researchers in the stamped addressed envelope provided. Those consenting to be interviewed were contacted by the research team to answer any questions they had and to schedule the interview.
Research ethical approval was granted by the Research Ethics Committee of the Faculty of Health Sciences in Trinity College Dublin and ethics committees associated with the mental health services participating in the EOLAS study.
Data analyses
Quantitative data were entered into the Statistical Package for the Social Sciences version 18 for analysis. To examine changes over time, Wilcoxon signed rank tests were conducted due to the small number of participants completing both pre- and post-EOLAS measures. For all inferential analyses, statistical significance was set at p<0.05.
All interviews were audio recorded and transcribed verbatim for thematic analysis. The qualitative data were entered into the software package NVivo version 8. Analysis involved listening to the audio recordings, while systematically coding the written transcripts for emerging themes and ideas. Codes were compared for similarity and differences, and merged into higher themes. The analytic approach was guided by the constant comparative process (Glaser & Strauss, Reference Glaser and Strauss1967). To enhance the rigour of the analysis, data were analysed by more than one person and findings compared. All interview participants were given codes to protect confidentiality.
Sample
Of the 30 who attended the first session of the EOLAS service user programme, 28 completed the pre-EOLAS questionnaires and 12 completed post-EOLAS questionnaires. Of the 12 with matched data and who provided information about their gender, the respondents were evenly split between males (50%; n=5) and females (50%; n=5). The age range was 30–63 years. All of these respondents attended over six EOLAS sessions: 30% (n=3) attended six, 20% (n=2) attended seven and 50% (n=5) attended eight.
All 25 family members who attended the first session of the EOLAS family member programme completed pre-EOLAS questionnaires and 18 completed post-EOLAS questionnaires. Of the 18 with matched data and who provided information about their gender, 76% (n=13) were female and 24% (n=4) were male. All were between 31 and 68 years of age. All respondents had attended four or more EOLAS sessions: 11% (n=2) attended four, 33% (n=6) attended five, 33% (n=6) attended six and 22% (n=4) attended seven (Table 2).
Table 2 Demographic profile of survey participants
A total of 19 programme participants participated in interviews (11 family members and eight service users). The family programme participants were mostly parents (n=7), split approximately evenly between mothers and fathers. There were also two siblings, one husband and one wife of user participants. In total, 11 facilitators participated in interviews: seven clinicians, two service users and two family members. Of the seven clinicians, there were roughly half males (n=3) and half females (n=4), all between the ages of 25–45 years. Of the service users, one was a 35-year old female and the other was a 27-year old male. The two family facilitators were both females in their 50s and were mothers of service users (see Table 3).
Table 3 Demographic profile of interview participants
Results
Impact of the programme on knowledge
Family members had a statistically significant increase in overall perceived levels of knowledge post-EOLAS [Wilcoxon z (7)=2.39, p<0.05]. Significantly greater knowledge among family members was found in relation to understanding their relatives’ legal rights in the mental health services [Wilcoxon z (12)=2.57, p<0.01], knowledge of how to help their relative deal with voices [Wilcoxon z (11)=2.23, p<0.05] and familiarity with strategies for helping their relative get involved in the local community [Wilcoxon z (12)=2.00, p<0.05]. Within the interviews, family members reported gaining a greater knowledge of medication and services available to people experiencing mental health difficulties:
That was the one that we just said, ‘If only we could have even spoken to a psychologist or whatever in the beginning,’ and that was fantastic and he [facilitator] discussed medications and discussed [other] things … so that night was the best (F 6).
Although it was not possible to examine changes in total perceived knowledge for service users as only two participants provided complete data, participants’ comments at interview indicated they had gained an enhanced understanding of their diagnosis, medication and developed a greater awareness of how to recognise and deal with stress:
The information was useful, especially about diagnosis and symptoms … It was the type of thing that was always fuzzy in my mind … I felt I picked up a lot from that … sort of put it all together (service user participant).
The one on medication was good … and I suppose the one the stress, you know the definitions of stress and how to recognise it, that type of stuff was good (service user participants).
While service user participants spoke of gaining more knowledge of medication, there was no statistically significant change in attitudes to drugs from pre- to post-programme.
One of the requests by service users and family members was to be given an opportunity as part of the programme to discuss issues with a psychiatrist. Service users and family members commented very favourably on this aspect of the programme and reported having received informative responses to their questions, which had enhanced their knowledge:
Consultant’s discussion excellent, gained great information, got answers from questions I didn’t know who to ask … simplified complicated things with good examples (service user feedback sheet).
The day the psychiatrist came to discuss was an absolute, oh, it was like doors being opened or something … that was fantastic and he discussed medications and discussed [other] things … so that night was the best (family member participant).
Some service users reported that the relapse prevention session was extremely useful as it enabled them to think about triggers and warning signs of diminished well-being as well as making them aware of the coping strategies and techniques that could be used in everyday life situations:
They tell you to relax … just focus on what you have to do, don’t be over doing it … if you think that you are going to be as high as a kite, … do your breathing exercises … and take it easy It helped me to understand myself better because I’d be really hard on myself (service user participant).
Impact of programme on self-advocacy
There were no statistically significant changes on the Self-Advocacy Scale for either family members. However, the comments indicated that service users in particular began to find their voice through the programme and were better able to ask questions and articulate their experiences:
You feel empowered in a couple of ways. One is to talk about your medication and another is to be more open about your thought process that’s causing the episodes … the triggers that causes it is very similar in each of my cases. I do feel that I can talk to the psychiatrist or the doctor about that [medication and triggers] now (service user participant).
Impact of programme on recovery attitudes and on hope
There were no statistically significant changes in recovery attitudes or views on hope for either family members or service users. Mixed accounts of hope and hopelessness might account for this lack of change. While some family members gleaned hope and a sense of optimism for the future through the group sessions, another participant reported that the potential longevity of medication in their loved one’s life was realised, thus creating a sense of despair. The two quotes below illustrate these opposing experiences:
I would say they would be brilliant at giving hope … at one stage I thought, there is no cure for schizophrenia and at one stage I thought terrible thoughts, ‘You’ll [referring to child] never be married, never have kids. I’ll never be a grandmother and all this kind of stuff,’ and I was put right back in my place and said, ‘Don’t ever say there’s no hope’ … I was told there are people who go out and get jobs … so I have to sort of stop that … it was all a good uplifting sort of, ‘Come on. Cop on’ and ‘It’s not like that at all’ (family member participant).
Well [name of psychiatrist guest speaker] came to talk to us about … and he gave a very interesting lecture and it was kind of starting to sink in with me then, because we were able to ask questions … I was thinking will he ever be able to come off the medication and I thought maybe someday … but now I see that that’s not going to be the case. [Name of service user] is going to be on this medication forever and a day (family member participant).
Impact of the programme on support
Although there were no statistically significant changes on the Social Network of Support Scale for family members a majority of participants (both service users and family members) felt that hearing other member’s stories was supportive (93%; n=28). The experience of meeting people with similar experiences and hearing other people’s stories enabled participants to learn from each other about how to deal with situations, to obtain mutual support and reduce their sense of isolation as well as resulting in the normalisation and validation of their experiences. The common experiences of participants meant that understanding was readily available and that experiences could be shared openly and honestly in a way that could not be done within other contexts.
You were able to talk about things that you couldn’t really talk to anyone else about because they don’t really understand (family member participant).
Central to this engagement was the atmosphere of trust and openness that was created at the outset of each programme through agreement on ground rules relating to confidentiality, respecting others’ opinions and allowing others to voice their views.
They [ground rules] were all sound like … You felt, you know, comfortable there (SU 5).
Perspectives on peer involvement and co-facilitation
The dual nature of the co-facilitation (i.e. by both a peer and clinician facilitator) was viewed positively, with 79% (n=23) of survey participants agreeing that having a peer as a facilitator on the programme was a positive experience. The remaining 21% (n=7) felt neutral about having a peer facilitator on the programme. Peers made a unique and distinct contribution by providing empathy and understanding as well as providing a source of hope and inspiration:
He [user facilitator] talked at your own level … You sort of think to yourself, ‘Well if they can do it and handle their illness like that, so can I. So you identify with that fact that they have actually taken that step … come into a meeting and say, ‘Yes, I suffer from schizophrenia … and now I’m coordinating this programme’ (service user participant).
There was a girl [family facilitator] … her [names relative] has schizophrenia … just nice to have her there because she’d experience … Someone who actually knows what it’s like to live with it (family member participant).
Clinical facilitators noted that working with the peer facilitators promoted empathy and enabled them to relate at a more ‘human’ level. One clinical facilitator described this change in the following way:
I found that [working with peer facilitators] really, really interesting, even those two days [of training], you know, because you weren’t talking about mental illness, you were talking about other things, on a more human level (CF 7).
Although it was clear that the presence of a peer facilitator was seen as a valuable aspect of the programme, some described how traditional power dynamics prevailed. For instance, one peer facilitator who described her working relationship with the clinical facilitator as ‘brilliant’ perceived this power differential through the symbolism of the clinician holding the keys to the EOLAS venue:
She [clinical facilitator] always had the keys (laughs). I actually laugh at these bundles of keys, you know, that people take around with them. It’s a sign of power (service user facilitator).
Peer facilitators themselves tended to over-value clinical expertise and spoke of lacking knowledge of illness, pharmacology and treatments; consequently, they tended to defer to the medical expertise of the clinicians.
I had my humanity and I had the experience of doing it [caring] but I didn’t have the medical background and that’s what I was lacking, more knowledge of that … (family facilitator).
Despite having a 2-day training, some clinician facilitators appeared to lack the facilitation skills necessary to enable co-facilitators to participate on an equal basis:
I found that there was a lack of clarity about the role of the co-facilitator … And it led to them kind of taking a step back … which is fine but then it led to the group being solely led by me and the co-facilitator not really having an input beyond welcoming people … (clinical facilitator).
Overall satisfaction with the programme
Nine out of 10 participants rated their satisfaction (90%; n=26) and enjoyment (93%; n=27) as high or extremely high, and all of the post-programme participants (n=26) would recommend the programme to others. Among both service users and family members, the majority (97%; n=29) felt that their views and opinions were respected (97%, n=29) and their knowledge was respected (87%, n=26). The programme was also evaluated positively in terms of its objectives and content, though family members did comment on the idealised nature of some of the content rather than reflecting the reality of how the mental health system operates. The majority (79%, n=22) of the sample felt that the planning sheets and learning materials were an effective way of learning and found the written information handouts very helpful. These positive outcomes were also reflected in comments made at interview:
I found it very helpful. I couldn’t find anything negative about it at all (family member participant).
I thought it was a very good course. It served its purpose (service user participant).
Discussion
The small number of participants who provided complete date restricted the statistical power of the analyses to find programme effects; in addition, the small sample size means that the findings may have limited generalisability. Consequently, the findings may be regarded as quite tentative until larger scale replication studies are complete. While a number of the questionnaires did not show statistically significant changes, findings from the interviews suggest that the programme had a number of positive outcomes, including increases in perceived knowledge, empowerment and support. Participants in both programmes spoke of having left the programme with a ‘better understanding’ of either their own or another’s mental health problem. Research has highlighted the supportive nature of peer groups (Rappaport, Reference Rappaport2000; Finn et al. Reference Finn, Bishop and Sparrow2007; Sibitz et al. Reference Sibitz, Amering, Gössler, Unger and Katschnig2007; Barber et al. Reference Barber, Rosenheck, Armstrong and Resnick2008; Resnick & Rosenheck, Reference Resnick and Rosenheck2008) and the present findings are consistent with this. Although the EOLAS programmes were primarily developed to provide information, they also appeared to meet a significant need for emotional support. Participants valued the opportunity to meet others in similar circumstances, share their experiences, learn from each other and provide mutual support. Family member participants went on to form two separate peer support groups, as a follow-up to their experience of the EOLAS programme.
One of the core differences between the EOLAS programmes and other programmes in the literature is the dual nature of the facilitation as it incorporated both peer and clinician facilitation. Overall, participants were positive about the dual nature of the facilitation and valued the role of each facilitator for different reasons. In their view, peers had credibility by virtue of self-experience and provided hope and inspiration. In contrast, the clinicians came with clinical expertise that they also valued. Having said this, findings indicate that on occasion, traditional power dynamics prevailed within the group, that is, with the clinical facilitators at times being positioned as the ‘leaders’ and with clinical knowledge being seen as more valuable than the voice of self-experience. As McDaid (Reference McDaid2006: 58) points out ‘simple equality of presence … will not ensure equal participation’, thus the over-referencing of professional expertise and the delegation of ‘trivial tasks’ to peers are issues that require urgent attention in subsequent facilitator training programmes. McDaid (Reference McDaid2006, 2009) advocates for greater capacity building among service users and practitioners, for example, by retraining professionals to value experiential knowledge and by rebalancing power relations through assigning authority to service users. Without this, there is a danger that participants leave the programme with a message that reinforces the lack of power that service users and their family members experience in effecting change within the mental health services. It may also deter service users and family members from becoming involved as facilitators in other projects for fear of tokenism.
Previous studies confine the delivery of programmes to people who had received a very specific diagnosis, with a reluctance to combine people from different diagnostic groups (Rummel et al. Reference Rummel, Hansen, Helbig, Pitschel-Walz and Kissling2005). In the early phase of the EOLAS project there were some concerns among clinicians around mixing people with different diagnoses. Practitioners were concerned that the ‘stigma’ attached to a diagnosis of schizophrenia may inhibit people who had a diagnosis of bipolar disorder attending, and that their information needs would differ. This study demonstrates that combining people with different diagnoses offers a promising approach, especially when there may not be enough potential participants or facilitators to make organising diagnosis-specific groups a feasible option. Findings from this study would also support Rabovsky & Stoppe’s (Reference Rabovsky and Stoppe2006) view that there are many aspects of information needs common to service users with a variety of diagnoses, including their desire for information on medication, diagnosis, service provision, advocacy, warning signs, family relationships and recovery.
Documenting how a programme was produced and making explicit the rationale for decisions is critical to implementing similar programmes across a diversity of settings, yet there is little available information on process with the literature (Park et al. Reference Park, Zafran, Stewart, Salsberg, Carolyn Ells, Rouleau, Estein and Valente2014). The collaborative nature of the project design, development, delivery and evaluation as described in the current two papers offers a blueprint for future developments and provides a potential framework by which other services can work towards achieving a number of the objectives of Vision for Change; such as involvement of service users and families as active participants in the ‘planning, development, delivery, evaluation and monitoring of mental health services’ (Department of Health and Children, Reference Borkin, Steffan, Krzton, Wishnick, Wilder and Yangarber2006: 91). It also offers a means of meeting the quality standards as laid down by the Mental Health Commission (2007: 32–40):
-
∙ standard 3.3, that ‘peer support/advocacy is available to service users of services’;
-
∙ standard 3.4, that ‘a clear accessible mechanism for participation in the delivery of mental health services is available to service users’; and
-
∙ standard 6.1, that ‘families, parents and carers are empowered as team members receiving information, advice and support as appropriate’.
The collaborative model of Participatory Action Research, which is underpinned by continuous cycles of action, understanding and evaluation, ensures that findings from the evaluation are fed into the next cycle of action. A key issue for consideration is the relationship between some of the measures and the content of the intervention. Some measures reflect quite general psychological constructs (e.g. recovery, hope and self-advocacy) and the item content of the measures may not have reflected the specific content of the intervention. Based on the feedback, a number of other specific changes have been implemented in both the content and process of the programmes. For example, training for facilitators has been both improved and extended, the emphasis on collaboration between facilitators has been emphasised, and user-friendly handbooks have been developed and are used in the next phase of roll out, which is currently being evaluated using a larger sample size.
Limitations
A core limitation relates to the fact that due to external pressures, the pilot phase of the study was under time pressure and consequently the programme may have been more didactic and less participative than desired. Participants volunteered to take part in the information programme and this may have attracted people who were more interested and positive about this form of initiative. In addition, the representativeness of those participants who volunteered for the evaluation interviews is unknown. The measures of knowledge and social support were generated by the research team to reflect the most salient issues relating to the content and process of EOLAS; additional psychometric validation is required to establish the validity and reliability of the scales. The overall numbers of participants who completed both the pre- and post-programme questionnaires was small, therefore extreme caution should be taken when interpreting any statistical findings, as the findings cannot be generalised beyond this study. The power of the study to find significant effects was compromised by the small number of participants. Finally, the study did not include a control group for comparison and long-term outcomes were not evaluated.
Conclusion
Overall, the EOLAS programmes are a step in the right direction towards making users of services and their families’ equal partners within the mental health services. Although further evaluation is ongoing, it is safe to say that it offers a novel template for communication and information sharing in a way that embodies the nature and principles of collaboration and offers service users and families a meaningful opportunity to become stakeholders in the mental health services with the knowledge and skills to inform others. Feedback from all stakeholders also indicates that the EOLAS project was a positive initiative within the service, commencing a dialogue on the importance of service user and family involvement and provides a template for balancing clinical expertise with experiential knowledge.
Acknowledgements
The authors report no conflicts of interest. The study was supported by a grant from GENIO.
