Nearly 90 percent of the population has at least one sibling (Reference Milevsky1;Reference Buist, Dekovic and Prinzie2). Little is known about the impact of sibling relationship on the development of and prognosis for individuals with first episode psychosis (FEP) (Reference Buist, Dekovic and Prinzie2). Although siblings of individuals with FEP are often the only age-appropriate peers and so the most natural agent to promote service users’ recovery, they are also vulnerable to mental ill health due to the negative impact of psychosis within the family (Reference Sin, Moone, Harris, Scully and Wellman3;Reference Smith, Greenberg and Mailick Seltzer4).
Conventional family and carer support resources are under-used by siblings who are in a busy phase of their own lives despite their active role and inputs in supporting their ill siblings (Reference Sin, Moone, Harris, Scully and Wellman3;5). Previous research has identified that siblings need information on psychosis and coping strategies coupled with peer support (Reference Sin6;Reference Sin, Jordan, Barley, Henderson and Norman7), delivered using modern information and communication technology (Reference Powell, Martin and Sutcliffe8). We developed such an internet-based intervention providing high quality information and multiple interactive components, using mixed methods staged within the Medical Research Council Complex Interventions Framework over an 18-month period (Reference Murray, Khadjesari and White9). The development of the internet intervention was informed by theoretical and empirical data from both systematic reviews on successful conventional interventions targeting family members (including siblings) (Reference Sin, Jordan, Barley, Henderson and Norman7;Reference Sin and Norman10) and focus group study with siblings (Reference Sin6) as well as an iterative participative consultation process (Reference Staley11;Reference Rotondi, Cucciare and Weingardt12).
We report here a usability study which aimed to evaluate user satisfaction and usability of the intervention-prototype. The E Sibling Project is the first known internet intervention dedicated for siblings of people with psychosis.
METHODS
The pilot usability study aimed to test out the usability, feasibility, and acceptability of the intervention-prototype. USERfit methodology which emphasizes that product development should be user-centered and driven by user requirements, was used to devise the usability evaluation (Reference Poulson, Ashby and Richardson13). The USERfit methodology guided the usability evaluation being conducted in a format of non-randomized user trial by a sample of twenty end-users (i.e., siblings of individuals with psychosis) (Reference Poulson, Ashby and Richardson13). Participants’ views on the prototype's social, practical, and product/ software usability were collected by means of an online usability evaluation questionnaire. Furthermore, usage data were collected and recorded by the monitoring system incorporated into Moodle (https://moodle.org/), the software used to build and run the online intervention. The focus of the analysis was to identify strengths and weaknesses of the prototype, and areas in need of further development and refinement in preparation to a larger scale trial. This pilot test was registered as part of a bigger trial at Current Clinical Trials (CCT unique registration number: ISRCTN01416694). Supplementary Figure 1, which can be viewed online at http://dx.doi.org/10.1017/S0266462314000488, shows the procedures of the usability test.
Advertisement and recruitment of the usability test was conducted online and offline through UK-based mental health charities who run support services for carers and families of people with severe mental illness (including Rethink Mental Illness, NHS Choices, Carers Trust) in England (Reference Sin, Henderson, Pinfold and Norman14). Siblings were eligible for the study if they were: aged 16 or above; had a brother or sister affected by psychosis with whom they have contact at least weekly on average over the previous three months; could read and understand English language in usual online communications; and had daily Internet access. Individuals who were unable to give informed consent or themselves affected by a mental illness that required treatment and care from a secondary/specialist mental health services were excluded. Potentially suitable and interested participants were screened for eligibility before informed consent was obtained, online. All participants were given a £10 goodwill online voucher to compensate for their time and contributions. The study was approved by the UK NHS Research Ethics Committee process (REC Approval reference number: 12/LO/1537).
Intervention
The E Sibling Project online intervention for siblings was designed and developed using an iterative process combining mixed research methods (15), that is, systematic reviews (Reference Sin, Jordan, Barley, Henderson and Norman7;Reference Sin and Norman10) and participative qualitative study with siblings in focus groups (Reference Sin6) to map out the essential ingredients of the desired intervention design and content and ongoing consultation with an Expert Advisory Group that comprised siblings, parents, individuals with personal experience of psychosis and mental health professionals (Reference Staley11). A detailed description of the development of the intervention and its format for use in the usability evaluative study and later the randomized controlled trial, has been published (Reference Sin, Henderson, Pinfold and Norman14). The Internet intervention has multiple components, as summarized in Table 1. Further details of the intervention can be viewed at http://siblingpsychosis.org/
Table 1: Summary of Content of the E Sibling Project Online Intervention
The participants were invited to use the intervention and all its components from May to June 2013 over a period of 4 weeks, from the date they had given informed consent to join the test and completed enrolment. We recommended that participants spend one to two hours per week using all components of the intervention, to self-pace use of the intervention, and pick and prioritize the content most relevant to their own needs and circumstances. Within the intervention, most of the education materials were readable as HTML documents or downloadable as PDF documents. The Siblings’ Blogs & Peer Forum and “Ask the Experts” spaces were interactive and participants were invited to write their own blogs and join in online discussion with peers (other participants) and post questions to the 12 professional members of the expert panel, including a GP, mental health nurses, a psychiatrist and mental health campaigners. A facilitator, a mental health nurse with over 10 years’ experience specializing in psychosocial interventions for people with psychosis and their families, moderated the online intervention daily during the week and posted weekly updates within the intervention online news forum to all participants with an aim to keep them engaged. On average, approximately two hours a working day was dedicated on facilitating the intervention as well as performing the administrative procedures involved in the usability test. A “Contact for support” function for either ICT support or emotional support was available for all participants.
Measures
The participants were asked to provide demographic data at baseline and to complete outcome measures questionnaires at two time-points: before they started accessing the online resource (week zero) and at the end of the test period (week five). In addition to outcome measures assessing siblings’ psychosocial well-being, knowledge and coping, participants were also invited to complete a post-intervention evaluation questionnaire (the results of which are reported in this study). The intervention materials were delivered and questionnaires completed online.
RESULTS
A total of 20 siblings, as required by the study design, were recruited over a three-week period from mid May 2013. Seven further siblings attempted to join the usability test but failed to meet all the inclusion criteria or give informed consent.
Nineteen of the 20 participants completed all the baseline assessments before using the intervention subsequently for four weeks; the other participant did not complete the baseline assessment hence did not access the online intervention. The nineteen usability test participants comprised three men and 16 women who had a sibling affected by psychosis. All participants were full biological siblings to their ill siblings. In terms of ethnicity groupings, fifteen participants were White (White British - 13 or White Other - 2); three were of mixed-race; and one was of Latin American heritage. The age range of participants was 20–58 years (mean = 35.4 years; SD = 9.6). All of the participants were in full-time or part-time work, education, or both. At the time of the study, most of the participants lived separately from their ill siblings, apart from one participant who was an older sister. Two participants were sisters themselves and reported demographic details of the same ill brother, hence the total number of service users involved was eighteen. The gender mix of service users differed from that of the sibling-participants; eleven were men and seven were women. The ages of ill siblings ranged from 20 to 52 years (mean = 33 years; SD = 8.5). Table 2 summarizes the demographics of the participants and their ill siblings. Supplementary Figure 1 shows the participant flow throughout the study.
Table 2. Demographics of the Participants and Their Ill Siblings Who Have Experiences of Psychosis
aTwo participants are siblings and both reported demographic details on the same ill sibling (brother).
Usage Data: Patterns of Use and Participation
Among the 17 participants who completed the full evaluation, their self-reported usage patterns suggested that most participants accessed and used the web site fortnightly (n = 11; 64.7 percent), whereas others used it nearly weekly (n = 5; 29.7 percent), and one participant used it 2–5 times a week (n = 1; 5.9 percent). Their average time spent on using the online intervention was estimated as 120 min per participant (SD = 72 min) over a four weeks duration (ranged from 30 to 300 min; median = 120 minutes).
The Moodle system usage statistics recorded the use of the various sections / components of the online intervention. In total, sixty-three unique visits/ logins by the nineteen participants were made over the usability test period. These visits ranged from relatively short 10- to 15-min spells to just under an hour slots. The average time of each visit was estimated as 25 min (SD = 12 min; median = 25 minutes). Most participants visited a few sections at each visit (or episode of login), thus generating multiple page-views per visit, culminating to the overall page-views figure of 906 for the whole online intervention over the usability test duration. Detailed usage data of each component of the online intervention is shown in Table 3 (these do not include usage data related to the baseline and post-test questions).
Table 3. E Sibling Project – Usage Data across Various Sections
Participants’ Evaluation of Usefulness and Acceptability
Of the 19 participants who tried out the intervention, 17 completed the post-usability test evaluation online. With regard to intervention content and its quality, all the participants (n = 17; 100 percent) evaluated the intervention highly and identified that the content was pitched at a level that was “just right” in terms of coverage of information, language use and quality. Just under 95 percent of all participants evaluated the intervention content as (very) relevant to them (n = 16). See Table 4 for further details. Commenting on the usefulness and relevance of the intervention, participants stated:
Table 4. E Sibling Project Online Intervention – Evaluation on Its Usefulness, Acceptability, and Usability
Note. The mode response to each evaluation item is highlighted in italic text and the optimal option to each evaluation item is highlighted in bold text.
-
• “Very helpful, informative and relevant.” (ESUser07, an older sister)
-
• “All the resources are relevant to someone in my situation.” (ESUser01, an older sister)
-
• “My sibling had his first psychotic episode 10 years ago, . . .. Had there been this sort of resource 10 years ago which provides information and guidance I think would have greatly reduced the distress I went through, . . . that others don't have to face the same experience.” (ESUser04, a younger sister)
As the online intervention provides interactive forums, one with a panel of experts (professionals) and the other with peers (other sibling-participants), we asked for specific feedback on these interactive functions. Ten of 17 participants liked the interactive functions and rated them, in particular the “Ask the Experts” forum, positively (n = 10, 58.8 percent). Participants’ feedback also favored the organization of the forums in which discussion topics were grouped into distinct categories along which discussion threads could be followed. One participant reported: “The best bit was the forum with the discussion starters (topics) as it's really good to start somewhere specific as it's such a wide and emotional topic . . . just would have liked to dialogue with others a bit more.” (ESUser11, an older sister) (See Tables 3 & 4 for further details). Many participants appreciated the “Ask the Experts” forum, commented that:
-
• “I was able to ask questions related to my sibling in a non-judgemental environment and where I was unable to ask elsewhere. . . I thought the forums were a really good idea and I gained most benefits from the expert forum.” (ESUser01, an older sister)
Participants’ Evaluation of Usability
We asked the participants to rate the online intervention in terms of its ease of use, presentation of materials, and logistics. With regard to logistics (i.e., How clearly laid out the resource materials was?), just over half of the participants evaluated the navigation around the materials as very or quite clear. However, three participants raised concerns about navigating through the site to find the materials they were looking for and suggested that a site map might help in illustrating the overall structure. In contrast, more participants found the flow of material (presentation) logical and easy to follow, than illogical.
Just over half of the participants found locating and downloading text-materials (very or quite) easy. However when it involved interactive text materials (e.g., interactive exercises within the “Promoting well-being toolkit” section), a quarter of participants reported that navigating out of the original text file they were in to complete the exercises and finding their way back could be confusing and sometimes unsuccessful. These navigation problems worsened when it came to navigating to audio-visual materials. Just over half of the participants did not even notice it or try it as the Web-links/ functions were not explicit enough within the main menu, and a minority found it difficult to access. (See Table 4 for further details.)
Overall, the participants found the online intervention quick and easy to use, despite the navigation problems identified above (See also Table 4). A majority of participants reported that it was very or quite quick for them to use the intervention, such as: to find the materials they want, download or print them, to submit answers and/or undertake interactive exercises. A few participants identified that the problems they experienced might be related to them using the intervention with a mobile device (such as a smart phone or tablet for which the software was not designed for) from a café or public venue that Internet access was comparatively slower. Most participants identified the intervention as very or quite easy to use.
Participants made various suggestions to make the online intervention “more intuitive (to use)”. These suggestions included: more instruction on how to interact on forums, better navigation between interactive elements, and a site-map to illustrate the structure of the web site.
Participants’ Overall Evaluation of the Online Intervention
Overall, the majority of participants found using the intervention helpful and just over three quarters of the participants (n = 13) would “definitely” or “probably” recommend it to other siblings. The remaining participants (n = 4) were “unsure” about recommending the intervention in its current format and design due to their concerns over its navigation and usability that, in their view, affected users’ experiences and hindered access to the content regardless the quality of the content (see Table 4). Participants’ evaluation on this aspect included:
-
• “Most definitely so helpful all the material in one place, and a safe place to ask experts and peers questions. Also so good at giving advice about looking after yourself as I didn't have any information re(garding) that at the time and as a result became quite poorly myself.” (ESUser 07, an older sister)
-
• “I can't think of a single reason why not (recommending this to other siblings). . . .. It gives you information, practical advice and access to forums where you can share emotional and practical experiences as well as experts to help clarify any medical concerns that frankly, no other website I have ever been on can with any credibility. It truly is a resource for siblings.” (ESUser02, an older brother)
DISCUSSION
Overall, the evaluative feedback from the participants was positive regarding the content and design of the online intervention. The feedback highlighted a high degree of usefulness and acceptability perceived by the target end-users, that is, siblings, on the online intervention. This match of expectation and experience validates the value of the iterative development process with active inputs from siblings, the lay public, service users, and mental health professionals in mapping out the essential ingredients and their interactions (Reference Sin6;Reference Staley11;Reference Pinfold, Duggan and Huxley16). “Ask the Experts” forum and Peer discussion Forum were the most well-used features, as well as two of the most desired. Moderation by qualified mental health professionals was appreciated in enhancing the credibility of the intervention (Reference Sin6).
Both recruitment and completion rates of the usability test exceeded our expectation and the averages of similar studies on e-mental health interventions (Reference Murray, Khadjesari and White9;Reference Koo and Skinner17–Reference Couper, Peytchev and Strecher20). Our recruitment rate of one in every three siblings we approached (healthy population in their 20–50s), was much higher than the above cited studies. This may be due to multiple recruitment strategies using both online (e.g., emails, online posts/ advertisement, our own Web site) and more conventional media, such as presentation at events/ conferences, promotion to clinicians who signposted potential participants to the usability test. These helped compensate for the usually low recruitment rate to similar studies despite high visits/ views resulting from using solely online recruitment strategies (Reference Koo and Skinner17–Reference Couper, Peytchev and Strecher20).
Our study participation rate (nineteen of twenty consented participants used the intervention, 95 percent) and completion rate (seventeen of twenty completed the post-use evaluation online, 85 percent) were also much higher than other similar studies investigating online intervention aiming to promote well-being in a general public population (Reference Powell, Hamborg and Stallard19;Reference Mitchell, Stanimirovic, Klein and Vella-Brodrick21). For instance, the recent trial of Powell et al. of a 12-week online cognitive-behavioral tool (called MoodGym) in UK reported a completion rate of 49.8 percent and a retention rate of 26.5 percent (Reference Powell, Hamborg and Stallard19). Our usability test lasted for four weeks (or 6 weeks inclusive of the pre and post-test measures) and hence might have suffered less attrition (Reference Eysenbach22). Moreover, the online goodwill (Amazon e-voucher of £10 value) payment was also well-received by the participants and might have arguably contributed to retention (Reference Khadjesari, Murray and Kalaitzaki23).
Usage of the online intervention (average = 2 hour over 4 weeks) was not as high as recommended (approximately an hour a week). Nonetheless, this is a common characteristic of e-health interventions as participants have the option to discontinue use easily or to use the system sparingly (Reference Powell, Hamborg and Stallard19;Reference Mitchell, Stanimirovic, Klein and Vella-Brodrick21;Reference Powell, Jennings and Armstrong24). In this usability test, the relatively short duration of four weeks may have affected the usage. There was higher usage in the latter half of the test period with obviously more frequent posts and exchanges on the forums. Some participants commented that they spent a week or two to get familiarized and orientated to the intervention, just “watching and lurking” in the beginning. Some would have liked a longer test period as they believed they would use the intervention much more with time and increased confidence.
The usage data recorded the number of visits to each component by the participants, hence arguably reflects the popularity of each section. However, caution should be exercised as evaluation of the usage data of each component needs to be considered together with the subjective feedback made by the participants to appraise how participants used and interacted with others in each component. For instance, out of the total 32 visits (or 224 page-views) made to the Peer Forum by the participants, only six posts were made by a few of them. Nonetheless, many participants fed back that they enjoyed reading others’ posts and that they had learned something and felt better understood by reading these posts without actively participated in the discussion.
The relatively low usage figure (81 visits/ 219 page-views) of the non-interactive sections (that include Information on psychosis, Further resources) and that of the interactive discussion forums, requires careful interpretation. Participants could easily download all the non-interactive information factsheets on one go and store them onto their own computer/ laptop and/ or mobile devices (e.g., tablets, smart phones) from where they could retrieve and read the factsheets without making any further login to our Web site. Indeed, some participants fed back that they really appreciated the function that allowed them to download materials as PDF files. However, participants needed to log in to visit the forums and this may account for the much higher usage data identified for the interactive forums which may also have a novelty element.
The usability study demonstrates that using an Internet-based information-giving and support intervention to promote well-being and coping is feasible and acceptable to siblings of people with psychosis. Further development on navigation and lay-out is needed and such refinement work, in accordance with the participants’ feedback and suggestions, has since been undertaken. The next step of the E Sibling Project will see the intervention being tested for its efficacy in impacting on siblings’ mental well-being, knowledge and coping with caring for their ill brother or sister through a randomized controlled trial. If proven to be effective, such a resource should be provided to siblings routinely and as early as possible once their brother or sister is known to the mental health service (Reference Milevsky1–Reference Sin, Moone, Harris, Scully and Wellman3). Psychoeducation for individuals affected by psychosis and their families/carers is one of the few highly recommended psychosocial interventions with a strong evidence-base to reduce relapse rate, improve knowledge and coping (Reference Xia, Merinder and Belgamwar25). The E Sibling Project online intervention has the potential to provide psychoeducation coupled with peer support to siblings to fit their dynamic service needs and busy lifestyle given its online design and delivery medium. We have optimized the potential of the intervention to be disseminated widely at minimal cost given its online delivery medium and English language being mainstream in many areas. Nonetheless, as the intervention emphasizes on signposting and networking resources in the local communities alongside virtual information-giving and support, wide dissemination following the RCT for its efficacy will require in-depth considerations for adaption and localization beyond the UK.
SUPPLEMENTARY MATERIAL
Supplementary Figure 1: http://dx.doi.org/10.1017/S0266462314000488
CONTACT INFORMATION
Jacqueline Sin, MSc, RNT, RMN, (Jacqueline.sin@kcl.ac.uk), NIHR Research Fellow, King's College London, James Clerk Maxwell Building, 57 Waterloo Road, London SE1 8WA
Claire Henderson, MRCPsych, PhD, Clinical Senior Lecturer in Psychiatry
Ian Norman, PhD, RN, Professor of Mental Health
CONFLICTS OF INTEREST
All authors declared that they have no conflict of interests.
