Hostname: page-component-745bb68f8f-b6zl4 Total loading time: 0 Render date: 2025-02-06T10:48:54.775Z Has data issue: false hasContentIssue false
  • English
  • Français

Inter-rater reliability of the Bereavement Risk Assessment Tool

Published online by Cambridge University Press:  04 May 2011

Caelin Rose ,
Wendy Wainwright ,
Michael Downing  and
Mary Lesperance
Caelin Rose*
Affiliation:
Victoria Hospice Society, Victoria, British Columbia, Canada
Wendy Wainwright
Affiliation:
Victoria Hospice Society, Victoria, British Columbia, Canada
Michael Downing
Affiliation:
Victoria Hospice Society, Victoria, British Columbia, Canada
Mary Lesperance
Affiliation:
University of Victoria, Victoria, British Columbia, Canada
*
Address correspondence and reprint requests to: Caelin Rose, Victoria Hospice Society, 1952 Bay St., Victoria, BC, V8R 1J8Canada. E-mail: caelin.rose@viha.ca
Rights & Permissions [Opens in a new window]

Abstract

Objective:

The Bereavement Risk Assessment Tool (BRAT) was designed to consistently communicate information affecting bereavement outcomes; to predict the risk for difficult or complicated bereavement based on information obtained before the death; to consider resiliency as well as risk; and to assist in the efficacy and consistency of bereavement service allocation. Following initial development of the BRAT's 40 items and its clinical use, this study set out to test the BRAT for inter-rater reliability along with some basic validity measures.

Method:

Case studies were designed based on actual patients and families from a hospice palliative care program. Bereavement professionals were recruited via the internet. Thirty-six participants assessed BRAT items in 10 cases and then estimated one of 5 levels of risk for each case. These were compared with an expert group's assignment of risk.

Results:

Inter-rater reliability for the 5-level risk scores yielded a Fleiss’ kappa of 0.37 and an intra-class correlation (ICC) of 0.68 (95% CI 0.5-0.9). By collapsing scores into low and high risk groups, a kappa of 0.63 and an ICC of 0.66 (95% CI 0.5-0.9) was obtained. Participant-estimated risk scores yielded a kappa of 0.24. Although opinion varied on the tool's length, participants indicated it was well organized and easy to use with potential in assessment and allocation of bereavement services. Limitations of the study include a small sample size and the use of case studies. Limitations of the tool include the subjectivity of some items and ambiguousness of unchecked items.

Significance of results:

The collapsed BRAT risk levels show moderately good inter-rater reliability over clinical judgement alone. This study provides introductory evidence of a tool that can be used both prior to and following a death and, in conjunction with professional judgment, can assess the likelihood of bereavement complications.

Keywords

Bereavement RiskBereavement AssessmentRisk FactorGriefComplicated Grief
Type
Original Articles
Information
Palliative & Supportive Care , Volume 9 , Issue 2 , June 2011 , pp. 153 - 164
Copyright
Copyright © Cambridge University Press 2011

Introduction

Risk assessment is seen as an important first step in determining the need for bereavement services and targeting services to those in most need (Aranda & Milne, Reference Aranda and Milne2000). The call for reliable and valid assessment tools is frequently cited in the literature as integral to a larger commitment to standards of bereavement care and hospice palliative care overall (Bromberg & Higginson, Reference Bromberg and Higginson1996; Keegan, Reference Keegan2002). To this end, the Bereavement Risk Assessment Tool, or BRAT, (Figure 1) was developed 1) to improve our ability to predict difficulties and complications for bereaved persons through timely and comprehensive assessment; 2) to standardize language and enhance communication among team members, and; 3) to improve the efficiency and consistency in the allocation of bereavement services.

Fig. 1. Bereavement Risk Assessment Tool (BRAT).

In hospice palliative care, the collection of personal, interpersonal, and circumstantial information of patients, caregivers, and family members begins prior to the death. Because it is not always possible to assess family members and their needs after the death, it can be particularly useful at the initiation of bereavement services when potential difficulties are already noted in a consistent and organized manner. Identifying concerns pre-death can also prompt earlier interventions that may benefit families and help them avoid some bereavement difficulties altogether.

Complicated Grief

The majority of current thinking tends to recognize grief as a multifaceted experience, encompassing complex interrelationships with other phenomena (Busch, Reference Busch2001; Stroebe et al., Reference Stroebe, Folkman, Hansson and Schut2006). It can be said that most people cope with the death of someone close to them reasonably well without any intervention (Aranda & Milne, Reference Aranda and Milne2000; Hogan, Greenfield, & Schmidt, Reference Hogan, Greenfield and Schmidt2001). For some, relatively simple actions such as the provision of information regarding normal grief experiences are sufficient (Boelen, van den Bout, & van den Hout, Reference Boelen, van den Bout and van den Hout2006). Approximately 10 to 20 percent of the bereaved population, however, are identified as having a more complicated grief reaction and may benefit from professional intervention (Prigerson & Jacobs, Reference Prigerson, Jacobs, Stroebe, Hansson, Stroebe and Schut2001; Stroebe, Schut & Stroebe, Reference Stroebe, Schut and Stroebe2007).

Recently, the term complicated grief has received more focused attention in the literature. Stroebe et al. (Reference Stroebe, Hansson, Stroebe and Schut2001) define complicated grief as the presence of a single or group of grief symptoms that deviate from the cultural norm in persistence and intensity and go on to consider nine identifying symptoms or factors. While intrusive or compelling thoughts of, or pining for, the deceased may feel devastating for a bereaved person, factors such as these are only considered significant when they persist over time or they significantly intrude on the person's daily life (Prigerson & Maciejewski, Reference Prigerson and Maciejewski2005).

Complicated grief has also been identified as a form of attachment disturbance which results in an unstable sense of self and relationship to others (Prigerson et al., Reference Prigerson, Shear and Frank1997). People who suffer from complicated grief are less likely to access professional help, placing them at even higher risk (Prigerson et al., Reference Prigerson, Silverman, Jacobs, Maciejewski, Kasl and Rosenheck2001). Prigerson and colleagues (Zhang, El-Jawahri, & Prigerson, Reference Zhang, El-Jawahri and Prigerson2006; Prigerson et al., Reference Prigerson, Horowitz, Jacobs, Parkes, Aslan and Goodkin2009) believe this classification of grief is clinically distinct from depression, anxiety, and Post Traumatic Stress Disorder (PTSD), although they suggest these phenomena may co-exist. It has further been proposed that complicated grief be renamed Prolonged Grief Disorder and be included in the 5th revision of the Diagnostic and Statistical Manual of Mental Disorders.

Bereavement Risk

Aranda and Milne (Reference Aranda and Milne2000) define bereavement risk as “the extent to which a person is susceptible to adverse outcomes associated with the loss of someone significant through death”. They further stipulate that “the identification of risk suggests the probability of adverse outcomes rather than an indication of cause and effect” (p. 8).

Stroebe et al. (Reference Stroebe, Folkman, Hansson and Schut2006) designed an integrative risk factor framework intended to improve the understanding of individual differences in adjustment to loss or death and the interaction between risk factors. They argue against examining any specific factor in isolation and suggest that the assessment of risk should include factors that mitigate future potential harm in the adjustment to loss. These protective factors acknowledge resiliencies and strengths and may suggest why people cope differently when experiencing similar circumstances. Unfortunately, few studies use an assessment model that includes both strengths and deficits (Aranda & Milne, Reference Aranda and Milne2000; Walsh-Burke, Reference Walsh-Burke2000) and consequently there is little empirical support as to the benefits of such an approach.

The BRAT

A selection of evidence-based research that highlights current thinking and understanding of bereavement risk and provides empirical support for all but 4 of the BRAT's risk indicators is listed in Table 1. These 4 items remain included in the tool based on the clinical experience of the team who developed the instrument. They are: “significant other with life-threatening illness/injury”; “cultural or language barriers to support”; "parent expresses concern regarding his/her ability to support child's grief"; and “significant anger with OUR hospice palliative care program”. The last indicator was included by the clinical team in order to quickly identify persons who expressed animosity towards the organization and who might consequently refuse support or believe that bereavement support would not be available to them as a result of their animosity. Further testing will need to address the validity and clinical benefit of these empirically unsubstantiated indicators as predictors of poor bereavement outcome.

Table 1. Support from the literature of BRAT's bereavement risk and positive outcome factors

Knowledge and understanding of the grief process, including bereavement risk, continues to be a relatively new field of study. In their review, Stroebe and Schut (Reference Stroebe, Schut and Neimeyer2001) write, “…there is still a great deal to criticize, and little to praise, in research on risk factors in bereavement outcome” (p. 366), and cite the absence of appropriate control groups as a major contributor. Not only are there disputes related to theoretical or professional paradigms (Archer, Reference Archer2001; Stroebe et al., Reference Stroebe, Hansson, Stroebe and Schut2001), but the role of the bereaved themselves in determining what constitutes risk is unclear (Hogan et al., Reference Hogan, Greenfield and Schmidt2001).

The question of how to assess risk in a consistent manner that is theoretically and methodologically sound remains outstanding (Stroebe, Stroebe, & Schut, Reference Stroebe, Stroebe and Schut2003; Center for the Advancement of Health, 2004). Also at question is whether we measure risk only to determine complicated grief as defined above, or to determine a predisposition for depression, anxiety, or other health related issues.

One study reports a clinical bias towards using risk assessment instruments developed in-house in 87% of the American hospice organizations it surveyed (Demmer, Reference Demmer2003). Although a number of tools have been developed more formally, there appear to be limitations for most. For example, some lack predictive utility for bereavement outcomes, (Theut et al., Reference Theut, Jordan, Ross and Deutsch1991; Jordan et al., Reference Jordan, Baker, Matteis, Rosenthal and Ware2005). Other tools fail to consider the circumstances of the death (Levy, Reference Levy1991), the presence of addictions or previous losses (Parkes & Weiss, Reference Parkes and Weiss1983; Kristjanson et al., Reference Kristjanson, Cousins, Smith and Lewin2005), or physical, social, religious, and psychological factors (Prigerson & Jacobs, Reference Prigerson, Jacobs, Stroebe, Hansson, Stroebe and Schut2001).

We suggest there is a need for a reliable and valid tool that measures potential for problems in bereavement, includes elements of risk and resiliency, provides an opportunity to identify these factors prior to the patient's death, and can be tailored to the resources and services of different organizations. This study begins the introduction of evidence with regard to the BRAT's inter-rater reliability along with a few foundational measures of validity.

Design and Method

The Tool

The BRAT is a 40-point list of both risk factors (36 items) and protective factors (4 items) that collectively predict the likelihood for difficulties or complications in bereavement. It was developed by a team of hospice palliative care clinicians comprising social workers, counsellors, psychologists, and child and youth specialists. Our psychosocial clinical team initially identified a list of factors believed to contribute to poor, exaggerated, or prolonged outcomes of grief. These factors were thematically organized and by consensus reduced to 36 independent items. Based on clinical experience, numerical values representing one of 5 levels of risk were assigned to each factor, reflecting greater or lesser relative risk. A range of total scores that identified 5 cumulative levels of risk was then identified. These levels of risk were designated as “no known risk” (level 1), “minimal risk” (level 2), “low risk” (level 3), “moderate risk” (level 4), and “high risk” (level 5)Footnote 1. To assist in the calculation of total scores, an Excel worksheet was developed to sum individual item scores and indicate the corresponding level of risk of the total score. Finally, the organizations’ bereavement team identified “default” servicesFootnote 2 to correspond with each of the BRAT's 5 levels of risk. Table 2 provides examples of how a bereavement program might offer services in relation to the risk levels identified by the BRAT.

Table 2. Examples of default bereavement service assignment matched with BRAT risk level

The tool was incorporated into clinical practice in 2003 and was revised in 2006 based on its use and an extensive review of the literature for both theoretical and empirical underpinnings. Some factors were re-worded for clarity, adjustments were made to the weighting of several factors and the 4 protective factors were added. A manual was developed to provide users with guidelines on using the tool and suggestions for incorporating it into a hospice palliative care bereavement program. Information on how to obtain a copy of the BRAT manual is available from Victoria Hospice (http://www.victoriahospice.org).

Methodology

The study design entailed asking participants to review 4 case studies containing 10 family members and caregivers of palliative patients. The case studies were created by 4 members of our clinical team with considerable experience in using the BRAT (hereafter referred to as the expert group). The case studies were based on actual patients and bereaved individuals and modified for anonymity. Thirty-one of the 40 BRAT itemsFootnote 3, as well as all 5 cumulative levels of risk, were represented in the descriptions of the 10 bereaved family members and their circumstances. The BRAT-determined risk levels were not revealed to the study participants; however, they were asked to provide a participant-estimated level of risk based on their own clinical experience using a 5-point scale identical to the risk levels of the BRAT. The cases were then tested in a pilot study and minor adjustments were made by the expert group to maximize the consistency of interpretation of the information provided.

Participants were recruited among social workers, professional counsellors, psychologists, and clergy who indicated they worked at least part-time with bereaved individuals. The Canadian Association of Social Workers, the Canadian Hospice Palliative Care Association, the British Columbia Hospice Palliative Care Association, and the National Hospice Palliative Care Organization (USA) were approached and agreed to forward an invitation email to their members and/or place a description of the study on their website. Those professionals who replied were mailed a study package by our research assistant. The package included a consent form, instructions on how to use the BRAT including a detailed explanation of each factor, the case studies, BRAT forms, as well as a questionnaire requesting demographic information and feedback on their use of the tool. Participants were informed their responses would remain anonymous and participation in any or all of the study was voluntary. A pre-paid postage return envelope was also provided. In order to provide a comparison to participant responses, the expert group also completed BRATs on the 10 bereaved family members.

Analysis

Descriptive statistics were used to describe the study sample. Reliability analyses were performed including intra-class correlations for absolute agreement of BRAT generated risk levels (McGraw & Wong, Reference McGraw and Wong1996). Inter-rater reliability of the individual items on the tool and the overall BRAT risk levels was assessed using Fleiss’ kappa, which is a generalization of Cohen's kappa to multiple raters (Fleiss, Reference Fleiss1971; Landis & Koch, Reference Landis and Koch1977). Fleiss’ kappa provides a conservative measure of agreement (Strijbos et al., Reference Strijbos, Martens, Prins and Jochems2006) over that which would be expected by chance. Values close to 1.0 indicate excellent agreement. Comments from the questionnaires were coded and grouped for common words or phrases by three members of the research team.

Results

Respondents

Fifty-four packages were sent out and 36 were returned. Of these, 18 were from Canada, 14 from the USA, and 4 were of unknown origin. Eleven respondents declined to indicate their age. Of the remaining 25, ages ranged from 31 to 74 years, with a mean of 52 (standard error 1.9) and median of 52. Additional participant demographics are provided in Table 3.

Table 3. Participant Demographics (N = 36)

Inter-rater Agreement

For the 31 BRAT items used in this study, values of Fleiss’ kappa ranged from 0.05 to 0.97. Six items had kappa values less than 0.4 (slight to fair agreement) and the remaining 25 items (81%) had kappa values above 0.4 (moderate to almost perfect agreement) (Landis & Koch, Reference Landis and Koch1977).

Table 4 provides the frequencies of the BRAT risk levels for each bereaved case. These BRAT risk levels demonstrated fair agreement with a kappa of 0.37 and an intra-class correlation of 0.68 (95% CI: 0.50–0.88).

Table 4. Frequency of BRAT risk levels by bereaved case including expert BRAT risk levels and participants’ percent agreement with the experts

Hypothesizing the participants’ BRAT risk levels would have higher agreement if using 2 levels of differentiation instead of 5, BRAT risk levels 1, 2 and 3 were collapsed into a Low risk category and levels 4 and 5 into a High risk category. Frequencies of percent agreement for Low/High risk levels are provided in Table 5. Fleiss’ kappa for the re-categorized data was 0.63, an improvement over that of the original scores. The intra-class correlation using 2 levels was 0.66 (95% CI: 0.47–0.87).

Table 5. Frequency of condensed* BRAT risk levels by bereaved case including the condensed expert BRAT risk levels and participants’ percent agreement with the experts

* Condensed BRAT risk levels condensed original BRAT levels 1, 2, and 3 into a Low category and BRAT levels 4 and 5 into a High category.

Validity

The original and collapsed BRAT risk levels were compared with the expert BRAT risk levels. Tables 2 and 3 also list percent agreement of participant BRAT risk levels with expert BRAT risk levels. The participant BRAT levels on the 5-point scale were in agreement with the expert levels 50 percent (average), whereas for the collapsed BRAT levels of Low/High, agreement was 87 percent (average). Participant-estimated levels of risk revealed only slight inter-rater agreement of 0.24 and ranged from 3 to 57 percent agreement with expert levels.

Thematic Analysis of Comments

Theme areas that arose in the analysis of questionnaire comments included ease of use, comprehensiveness, application and usefulness related to clinical practice, and limitations. Participants indicated the BRAT to be well organized and easy to use although there was a range of opinion regarding its length. Some participants found it quick and appreciated a one-page tool, while others found it long but wondered how to maintain its comprehensiveness in a shorter format. Many appreciated the inclusion of both protective and risk factors, along with the inclusion of specific areas of assessment, including spirituality, suicide risk, and culture.

Regarding clinical practice, the tool was thought to be useful for assessment and service delivery as well as training and advocacy. Participants acknowledged that the tool could not replace clinical judgment, but would be a resource or guiding instrument for the professional. A few participants thought they would use the tool as an adjunct resource in their practice or to supplement an existing tool.

There were a number of comments that reflected limited understanding and knowledge of the BRAT and its use, including questions about the calculation of risk levels, weighting of items and the definition of certain factors. Perceived limitations of the BRAT included subjectivity in identifying some factors, possible missing risk or protective factors and, as mentioned earlier, the large number of factors.

Discussion

The BRAT was designed with the following in mind: to assess and communicate information affecting bereavement; to predict the risk for difficult or complicated grief following death based on assessment prior to the death; to consider relevant resiliencies; and to assist in the efficacy and consistency of bereavement service allocation. The tool differentiates degrees of risk, considers the interaction of variables, and is amenable to the changing nature of the grief process. In these intentions, it seems the BRAT is somewhat unique.

The primary goal of this study was to estimate the inter-rater reliability of the tool. Using case studies, 36 psychosocial professionals with limited familiarity with the tool completed BRATs on 10 family members and caregivers of hospice palliative care patients. Perhaps due to this lack of familiarity, we found only fair agreement (kappa = 0.37) in their identification of levels of risk. The kappa value of agreement for factor selection was more encouraging, with 81 percent demonstrating moderate and above agreement. When risk levels were collapsed from 5 to 2, the kappa increased significantly to 0.63, indicating greater consistency in identifying low and high risk. Similarly, Robinson et al. (Reference Robinson, Nuamah, Lev and McCorkle1995), when evaluating the 8-item Bereavement Risk Index (BRI) which was designed to differentiate between low, moderate, and high levels of risk, also found discrimination between low and high levels only.

A noteworthy finding was that there was poor agreement for the participant-estimated level of risk based on their clinical judgement alone. It is possible that this particular group of professionals weighted these risk factors differently than the clinical team that developed the tool, or, more importantly, this may speak to the complexity of determining bereavement risk without the use of any tool and the potential for a wide interpretation of who is at risk without an anchoring structure.

The length and complexity of the tool poses several challenges. Some participants, in completing the BRAT, failed to include relevant information from the case studies. We hypothesize that keeping track of 40 potential indicators may be overwhelming for a new user and, therefore, would expect reliability to increase if these same professionals were to have further experience and familiarity with the tool.

As seen in Table 4, participant BRAT risk levels were in low agreement with the expert group in two cases (#5 and #8) with only 9 and 3 percent agreement, respectively. These cases involved a higher number of indicators causing us to speculate that lower agreement may be due to the complexity of these cases and the potential to miss or “sweep over” relevant indicators. This may also be the case for instances when there was a wide variance of risk, such as in case #9.

Some factors were simply misinterpreted. For example, one case study included a bereaved adult who had lost a parent. Death of a parent when the bereaved is an adult is not a BRAT item, yet, when completing the BRAT for this case, 2 participants marked the item “death of a parent, parental figure or sibling” which is listed under the heading “Children & Youth”. It appeared some participants also mistakenly identified factors related to the patient rather than to the bereaved.

A further limitation of the tool is the interpretation of unmarked indicators. A blank checkbox could mean the indicator is absent or that it has not been assessed. This ambiguity might be addressed by placing a checkmark beside the indicators that are present, marking an “X” beside the factors that are assessed but absent, and leaving the checkbox empty if the factor was not assessed. Although we believe bereavement assessment should attempt to include all of the BRAT's factors – not as a checklist per se, but as part of the overall intake and interaction with family members and caregivers during the care of the patient - this is not always achieved or possible. As an example, situations when patients are referred to a palliative care service only days prior to their death provides little opportunity to get to know family members and record pertinent bereavement risk information. We acknowledge that assessment is limited to information observed, solicited, or offered, and that our knowledge about clients at the best of times is imperfect and unfolding. Consequently, the BRAT is more likely to underestimate than overestimate risk.

The BRAT does not measure deeper psychological phenomena such as cognitive patterns, personality structure, or attachment styles, each of which have been identified as indicators of risk (Boelen, van den Bout, & van den Hout, Reference Boelen, van den Bout and van den Hout2003; Onrust & Cuijpers, Reference Onrust and Cuijpers2006; Schnider, Elhai, & Gray, Reference Schnider, Elhai and Gray2007). However, it can be used in conjuction with other tools that do measure these phenomena.

Limitations of the research study itself include the small sample size which affected our ability to statistically discriminate between 5 levels of risk. By not including 9 of the tool's 40 risk factors within the case studies, we did not acheive a complete testing of the tool, and the use of case studies rather than actual clients created some artificiality to the process. Although it would require a much more sophisticated and ethically sensitive methodology, testing the BRAT prospectively within a hospice palliative care program would be ideal. However, given some of the 40 items occur infrequently in practice, this may impede accumulation of a sufficient sample involving all of BRAT's items.

A review of the number of the BRAT's items seems warranted, including the 4 items that do not appear to have empirical support. However, in order to be useful to bereavement clinicians, we believe any reduction in the number of items must be weighed against the need for accuracy and comprehensiveness.

The literature and our experience both suggest there are many factors to be considered when assessing bereavement risk. While assessment is complex, it is critical in identifying those who may benefit from various kinds of bereavement support. We have provided evidence that a bereavement risk assessment tool offers advantages in identifying individuals at risk over clinical judgement alone. Some current tools, including the BRAT, show moderate reliability in assessing risk but only within wide parameters (low versus high). Multiple limitations of most of these tools remain. Future studies are needed in refining such tools, including their accuracy in identifying outcomes and best practices.

ACKNOWLEDGMENTS

This study was funded in part by grants from the Canadian Institutes of Health Research New Emerging team, the Victoria Foundation and the Victoria Hospice Society. We wish to thank the psychosocial team at Victoria Hospice and our research assistants, Stacey Slager and Jan Walker, for their contributions in this study.

Footnotes

1 It is understood that in speaking of bereavement risk, we are referring to known risk. We assume that information collected is incomplete and that it changes over time.

2 The term “default” refers to services initiated by the organization such as mail-outs and scheduled phone calls but does not include—or preclude—a bereaved person‘s request for additional services such as individual or group counselling.

3 Only 27 of the 36 “risk” items were used to construct the bereaved cases as it was not possible to designate all 36 risk items to 10 cases and achieve 2 cases of each of 5 levels of risk. All 4 “positive outcome” items, however, were used.

References

REFERENCES

Ajdacic-Gross, V., Ring, M., Gadola, E., Lauber, C., Bopp, M., Gutzwiller, F., et al. (2008). Suicide after bereavement: An overlooked problem. Psychological Medicine, 38(5), 673–676.CrossRefGoogle ScholarPubMed
Aranda, S. & Milne, D. (2000). Guidelines for the Assessment of Complicated Bereavement Risk in Family Members of People Receiving Palliative Care. Melbourne: Centre for Palliative Care.Google Scholar
Archer, J. (2001). Broad and narrow perspectives in grief theory: Comment on Bonanno and Kaltman (1999). Psychological Bulletin, 127(4), 554–560.CrossRefGoogle ScholarPubMed
Barrera, M., D'Agostino, N.M., Schneiderman, G., Tallett, S., Spencer, L. & Jovcevska, V. (2007). Patterns of parental bereavement following the loss of a child and related factors. Omega: Journal of Death and Dying, 55(2), 145–167.CrossRefGoogle ScholarPubMed
Barry, L.C., Kasl, S.V. & Prigerson, H.G. (2002). Psychiatric disorders among bereaved persons: The role of perceived circumstances of death and preparedness for death. The American Journal of Geriatric Psychiatry: Official Journal of the American Association for Geriatric Psychiatry, 10(4), 447–457.CrossRefGoogle ScholarPubMed
Bass, D.M., Bowman, K. & Noelker, L.S. (1991). The influence of caregiving and bereavement support on adjusting to an older relative's death. The Gerontologist, 31(1), 32–42.CrossRefGoogle Scholar
Boelen, P., van den Bout, J. & van den Hout, M. (2006). Negative cognitions and avoidance in emotional problems after bereavement: A propective study. Behavioural Research and Therapy, 44, 1657–1672.CrossRefGoogle Scholar
Boelen, P.A., van den Bout, J. & van den Hout, M.A. (2003). The role of negative interpretations of grief reactions in emotional problems after bereavement. Journal of Behavioral Therapy and Experimental Psychiatry, 34(3–4), 225–238.CrossRefGoogle ScholarPubMed
Bonanno, G.A., Wortman, C.B., Lehman, D.R., Tweed, R.G., Haring, M., Sonnega, J., et al. (2002). Resilience to loss and chronic grief: A prospective study from preloss to 18-months postloss. Journal of Personality and Social Psychology, 83(5), 1150–1164.CrossRefGoogle ScholarPubMed
Boscalglia, N. & Clarke, D. (2007). Sense of coherence as a protective factor for demoralization in women with a recent diagnosis of gynaecological cancer. Psycho-Oncology, 16, 189–195.CrossRefGoogle Scholar
Brazil, K., Bédard, M. & Willison, K. (2003). Bereavement adjustment and support among caregivers. Journal of Mental Health and Aging, 9(3), 193–204.Google Scholar
Brent, D., Melhem, N., Donohoe, M.B. & Walker, M. (2009). The incidence and course of depression in bereaved youth 21 months after the loss of a parent to suicide, accident, or sudden natural death. American Journal of Psychiatry, 166(7), 786–794.CrossRefGoogle ScholarPubMed
Bromberg, M.H. & Higginson, I. (1996). Bereavement follow-up: What do palliative support teams actually do? Journal of Palliative Care, 12(1), 12–17.CrossRefGoogle Scholar
Busch, C.J. (2001). On creating a healing story: One chaplain's reflections on bereavement and grief. Innovations in End-of-Life Care, 3, 3.Google Scholar
Byrne, G.J.A. & Raphael, B. (1997). The psychological symptoms of conjugal bereavement in elderly men over the first 13 months. International Journal of Geriatric Psychiatry, 12(2), 241–251.3.0.CO;2-0>CrossRefGoogle ScholarPubMed
Byrne, G.J.A., Raphael, B. & Arnold, E. (1999). Alcohol consumption and psychological distress in recently widowed older men. Australian and New Zealand Journal of Psychiatry, 33(5), 740–747.CrossRefGoogle ScholarPubMed
Center for the Advancement of Health (2004). Report on bereavement and grief research. Death Studies, 28, 491–575.CrossRefGoogle Scholar
Cerel, J., Fristad, M.A., Verducci, J., Weller, R.A. & Weller, E.B. (2006). Childhood bereavement: Psychopathology in the 2 years postparental death. Journal of the American Academy of Child and Adolescent Psychiatry, 45(6), 681–690.CrossRefGoogle ScholarPubMed
Chentsova-Dutton, Y., Shucter, S., Hutchin, S., Strause, L., Burns, K., Dunn, L., et al. (2002). Depression and grief reactions in hospice caregivers: From pre-death to 1 year afterwards. Journal of Affective Disorders, 69(1), 53–60.CrossRefGoogle ScholarPubMed
Christ, G., Bonanno, G.A., Malkinson, R. & Rubin, S. (2003). Bereavement experiences after the death of a child. In Field, M. & Berhman, R. (Eds.), When Children Die: Improving Palliative and End-of-Life Care for Children and Their Familes. Washington, DC: National Academy Press.Google Scholar
Currier, J.M., Holland, J.M. & Neimeyer, R.A. (2006). Sense-making, grief, and the experience of violent loss: Toward a mediational model. Death Studies, 30(5), 403–428.CrossRefGoogle Scholar
Demmer, C. (2003). A national survey of hospice bereavement services. Omega: Journal of Death and Dying, 47(4), 327–341.CrossRefGoogle Scholar
Dodd, P., Dowling, S. & Hollins, S. (2005). A review of the emotional, psychiatric and behavioural responses to bereavement in people with intellectual disabilities. Journal of Intellectual Disability Research, 49(7), 537–543.CrossRefGoogle ScholarPubMed
Dodd, P., Guerin, S., McEvoy, J., Buckley, S., Tyrrell, J. & Hillery, J. (2008). A study of complicated grief symptoms in people with intellectual disabilities. Journal of Intellectual Disability Research, 52(5), 415–425.CrossRefGoogle ScholarPubMed
Doka, K. (Ed.). (1996). Living with Grief After Sudden Loss. Washington, DC: Hospice Foundation of America.Google Scholar
Duberstein, P.R., Conwell, Y. & Cox, C. (1998). Suicide in widowed persons: A psychological autopsy comparison of recently and remotely bereaved older subjects. The American Journal of Geriatric Psychiatry: Official Journal of the American Association for Geriatric Psychiatry, 6(4), 328–334.CrossRefGoogle Scholar
Dumont, I., Dumont, S. & Mongeau, S. (2008). End-of-life care and the grieving process: Family caregivers who have experienced the loss of a terminal-phase cancer patient. Qualitative Health Research, 18(8), 1049–1061.CrossRefGoogle ScholarPubMed
Dyregrov, K., Nordanger, D. & Dyregrov, A. (2003). Predictors of psychosocial distress after suicide, SIDS and accidents. Death Studies, 27(2), 143–165.CrossRefGoogle ScholarPubMed
Espinosa, J. & Evans, W.N. (2008). Heightened mortality after the death of a spouse: Marriage protection or marriage selection? Journal of Health Economy, 27(5), 1326–1342.CrossRefGoogle ScholarPubMed
Field, N.P. & Sundin, E.C. (2001). Attachment style in adjustment to conjugal bereavement. Journal of Social & Personal Relationships, 18(3), 347–361.CrossRefGoogle Scholar
Fleiss, J. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.CrossRefGoogle Scholar
Gamino, L. & Sewell, K. (2004). Meaning constructs as predictors of bereavement adjustment: A report from the Scott & White Grief Study. Death Studies, 28, 397–421.CrossRefGoogle ScholarPubMed
Gamino, L.A., Sewell, K.W. & Easterling, L.W. (2000). Scott and White grief study Phase 2: Toward an adaptive model of grief. Death Studies, 24(7), 633–660.Google Scholar
Goldsmith, B., Morrison, R.S., Vanderwerker, L.C. & Prigerson, H.G. (2008). Elevated rates of prolonged grief disorder in African Americans. Death Studies, 32(4), 352–365.CrossRefGoogle ScholarPubMed
Greeff, A. & Human, B. (2004). Resilience in families in which a parent has died. American Journal of Family Therapy, 32, 27–42.CrossRefGoogle Scholar
Grimby, A. & Johansson, A.K. (2009). Factors related to alcohol and drug consumption in Swedish widows. American Journal of Hospice & Palliative Care, 26(1), 8–12.CrossRefGoogle ScholarPubMed
Hays, J.C., Gold, D.T. & Pieper, C.F. (1997). Sibling bereavement in late life. Omega: Journal of Death and Dying, 35(1), 25–42.CrossRefGoogle Scholar
Hebert, R., Dang, Q. & Schulz, R. (2006). Preparedness for the death of a loved one and mental health in bereaved caregivers of dementia patients: Findings from the REACH study. Journal of Palliative Medicine, 9, 683–693.CrossRefGoogle ScholarPubMed
Hebert, R.S., Dang, Q. & Schulz, R. (2007). Religious beliefs and practices are associated with better mental health in family caregivers of patients with dementia: Findings from the REACH study. American Journal of Geriatric Psychiatry, 15(4), 292–300.CrossRefGoogle ScholarPubMed
Hogan, N.S., Greenfield, D.B. & Schmidt, L.A. (2001). Development and validation of the Hogan Grief Reaction Checklist. Death Studies, 25(1), 1–32.CrossRefGoogle ScholarPubMed
Horowitz, M.J., Siegel, B., Holen, A. & Bonanno, G.A. (1997). Diagnostic criteria for complicated grief disorder. American Journal of Psychiatry, 154(7), 904–910.Google ScholarPubMed
Johnson, J.G., Zhang, B., Greer, J.A. & Prigerson, H.G. (2007). Parental control, partner dependency, and complicated grief among widowed adults in the community. Journal of Nervous and Mental Disorders, 195(1), 26–30.CrossRefGoogle ScholarPubMed
Jonasson, J.M., Hauksdottir, A., Valdimarsdottir, U., Furst, C.J., Onelov, E. & Steineck, G. (2009). Unrelieved symptoms of female cancer patients during their last months of life and long-term psychological morbidity in their widowers: A nationwide population-based study. European Journal of Cancer, 45(10), 1839–1845.CrossRefGoogle ScholarPubMed
Jordan, J.R., Baker, J., Matteis, M., Rosenthal, S. & Ware, E.S. (2005). The Grief Evaluation Measure (GEM): An initial validation study. Death Studies, 29(4), 301–332.CrossRefGoogle ScholarPubMed
Keegan, O. (2002). Standards for bereavement care: Irish Hospice Foundation, Paper No. 2. Retrieved from http://www.hospice-foundation.ie/cfmdocs/pdf/standards_bereavement_care.pdf.Google Scholar
Keesee, N.J., Currier, J.M. & Neimeyer, R.A. (2008). Predictors of grief following the death of one's child: The contribution of finding meaning. Journal of Clinical Psychology, 64(10), 1145–1163.CrossRefGoogle ScholarPubMed
Kelly, B., Edwards, P., Synott, R., Neil, C., Baillie, R. & Battistutta, D. (1999). Predictors of bereavement outcome for family carers of cancer patients. Psycho-Oncology, 8(3), 237–249.3.0.CO;2-C>CrossRefGoogle ScholarPubMed
Kersting, A., Kroker, K., Horstmann, J., Ohrmann, P., Baune, B., Arolt, V., et al. (2009). Complicated grief in patients with unipolar depression. Journal of Affective Disorders, 118, 201–204.CrossRefGoogle ScholarPubMed
Kim, S.H. (2009). The influence of finding meaning and worldview of accepting death on anger among bereaved older spouses. Aging & Mental Health, 13(1), 38–45.CrossRefGoogle ScholarPubMed
Kissane, D., Bloch, S. & McKenzie, D. (1997). Family coping and bereavement outcome. Palliative Medicine, 11, 191–201.CrossRefGoogle ScholarPubMed
Kristjanson, L.J., Cousins, K., Smith, J. & Lewin, G. (2005). Evaluation of the Bereavement Risk Index (BRI): A community hospice care protocol. International Journal of Palliative Nursing, 11(12), 610.CrossRefGoogle ScholarPubMed
Landis, J.R. & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.CrossRefGoogle ScholarPubMed
Levy, L.H. (1991). Anticipatory grief: Its measurement and proposed reconceptualization. Hospice Journal, 7(4), 1–28.CrossRefGoogle ScholarPubMed
Li, L. (2005). From caregiving to bereavement: Trajectories of depressive symptoms among wife and daughter caregivers. Journal of Gerontology, 60B(4), 190–198.CrossRefGoogle Scholar
Lin, K., Sandler, I., et al. (2004). Resilience in parentally bereaved children and adolescents seeking preventive services. Journal of Clinical Child and Adolescent Psychology, 33(4): 673–683.CrossRefGoogle ScholarPubMed
Lundin, T. (1984). Morbidity following sudden and unexpected bereavement. The British Journal of Psychiatry: The Journal of Mental Science, 144, 84–88.CrossRefGoogle ScholarPubMed
Lusyne, P. & Page, H. (2008). The impact of children on a parent's risk of suicide following death of a spouse, Belgium 1991–96. Population Studies, 62(1), 55–67.CrossRefGoogle ScholarPubMed
MacHale, R. & Carrey, S. (2002). An investigation of the effects of bereavement on mental health and challenging behaviour in adults with learning disability. British Journal of Learning Disabilities, 30, 113–117.CrossRefGoogle Scholar
McGraw, K. & Wong, S. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.CrossRefGoogle Scholar
Melhem, N., Day, N., Shear, M., Day, R., Reynolds, C.I. & Brent, D. (2003). Predictors of complicated grief among adolescents exposed to a peer's suicide. Journal of Loss & Trauma - Special issue: Risk and Resiliency Following Trauma and Traumatic Loss, 9(1), 21–34.CrossRefGoogle Scholar
Melhem, N.M., Walker, M., Moritz, G. & Brent, D.A. (2008). Antecedents and sequelae of sudden parental death in offspring and surviving caregivers. Archives of Pediatric and Adolescent Medicine, 162(5), 403–410.CrossRefGoogle ScholarPubMed
Mitchell, A.M., Kim, Y., Prigerson, H.G. & Mortimer, M.K. (2005). Complicated grief and suicidal ideation in adult survivors of suicide. Suicide and Life-Threatening Behavior, 35(5), 498–506.CrossRefGoogle ScholarPubMed
Moskowitz, J.T., Folkman, S. & Acree, M. (2003). Do positive psychological states shed light on recovery from bereavement? Findings from a 3-year longitudinal study. Death Studies, 27(6), 471–500.CrossRefGoogle ScholarPubMed
Nolen-Hoeksema, S. & Larson, J. (1999). Coping with Loss. Mahwah, NJ: Erlbaum.Google Scholar
O'Connor, M.F., Wellisch, D.K., Stanton, A.L., Eisenberger, N.I., Irwin, M.R. & Lieberman, M.D. (2008). Craving love? Enduring grief activates brain's reward center. Neuroimage, 42(2), 969–972.CrossRefGoogle ScholarPubMed
Onrust, S.A. & Cuijpers, P. (2006). Mood and anxiety disorders in widowhood: A systematic review. Aging & Mental Health, 10(4), 327–334.CrossRefGoogle ScholarPubMed
Ott, C.H. (2003). The impact of complicated grief on mental and physical health at various points in the bereavement process. Death Studies, 27(3), 249–272.CrossRefGoogle ScholarPubMed
Parkes, C. & Weiss, R. (1983). Recovery From Bereavement. New York: Basic.Google Scholar
Prigerson, H., Horowitz, M., Jacobs, S., Parkes, C., Aslan, M., Goodkin, K., et al. (2009). Prolonged grief disorder: Psychometric validation of criteria proposed for DSM-V and ICD-11. PLoS Medicine, 6(8), e1000121.CrossRefGoogle ScholarPubMed
Prigerson, H. & Jacobs, S. (2001). Traumatic grief as a distinct disorder: A rationale, concensus criteria, and a preliminary empirical test. In Stroebe, M., Hansson, R., Stroebe, W. & Schut, H. (Eds.), Handbook of Bereavement Research: Consequences, Coping and Care. Washington, DC: American Psychological Association. pp 613–645CrossRefGoogle Scholar
Prigerson, H. & Maciejewski, P. (2005). A call for sound empirical testing and evaluation of criteria for complicated grief proposed for DSM-V. Omega: Journal of Death and Dying, 52(1), 9–19.CrossRefGoogle Scholar
Prigerson, H., Shear, M., Frank, E., et al. (1997). Traumatic grief: A case of of loss-induced trauma. American Journal of Psychiatry, 154, 1003–1009.Google ScholarPubMed
Prigerson, H., Silverman, G., Jacobs, S., Maciejewski, P., Kasl, S. & Rosenheck, R. (2001). Disability, traumatic grief, and the underutilization of health services. Primary Psychiatry, 8, 61–69.Google Scholar
Prigerson, H.G., Cherlin, E., Chen, J.H., Kasl, S.V., Hurzeler, R. & Bradley, E.H. (2003). The stressful caregiving adult reactions to experiences of dying (SCARED) scale: A measure for assessing caregiver exposure to distress in terminal care. American Journal of Geriatric Psychiatry, 11(3), 309–319.CrossRefGoogle Scholar
Robinson, L.A., Nuamah, I.F., Lev, E. & McCorkle, R. (1995). A prospective longitudinal investigation of spousal bereavement examining Parkes and Weiss' bereavement risk index. Journal of Palliative Care, 11(4), 5–13.CrossRefGoogle Scholar
Rubin, S., Nadav, O., Malkinson, R., Koren, D., Goffer-Shnarch, M. & Michaeli, E. (2009). The Two-Track Model of Bereavement Questionaire (TTBQ): Development and validation of a relational measure. Death Studies, 33, 305–333.CrossRefGoogle Scholar
Sanders, S., Ott, C.H., Kelber, S.T. & Noonan, P. (2008). The experience of high levels of grief in caregivers of persons with Alzheimer's disease and related dementia. Death Studies, 32(6), 495–523.CrossRefGoogle ScholarPubMed
Schnider, K., Elhai, J. & Gray, M. (2007). Coping style use predicts posttraumatic stress and complicated grief symptom severity among college students reporting a traumatic loss. Journal of Counseling Psychology, 54(3), 344–350.CrossRefGoogle Scholar
Siegel, J.M. & Kuykendall, D.H. (1990). Loss, widowhood, and psychological distress among the elderly. Journal of Consulting and Clinical Psychology, 58(5), 519–524.CrossRefGoogle ScholarPubMed
Silverman, G., Johnson, J. & Prigerson, H. (2001). Preliminary explorations of the effects of prior trauma and loss on risk for psychiatric disorders in recently widowed people. Israel Journal of Psychiatry and Related Sciences, 38, 202–215.Google ScholarPubMed
Strijbos, J., Martens, R., Prins, F. & Jochems, W. (2006). Content analysis: What are they talking about? Computers & Education, 46, 29–48.CrossRefGoogle Scholar
Stroebe, M., Hansson, R., Stroebe, W. & Schut, H. (Eds.). (2001). Handbook of Bereavement Research: Consequences, Coping and Care. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
Stroebe, M. & Schut, H. (2001). Meaning making in the dual process model of coping of bereavement. In Neimeyer, R. (Ed.), Meaning Reconstruction and the Experience of Loss. Washington, DC: American Psychological Association.Google Scholar
Stroebe, M., Schut, H. & Stroebe, W. (2005). Attachment in coping with bereavement: A theoretical integration. Review of General Psychology, 9(1), 48–66.CrossRefGoogle Scholar
Stroebe, M., Schut, H. & Stroebe, W. (2007). Health outcomes of bereavement. The Lancet, 370, 1960–1973.CrossRefGoogle ScholarPubMed
Stroebe, M., Stroebe, W. & Schut, H. (2003). Bereavement research: Methodological issues and ethical concerns. Palliative Medicine, 17(3), 235–240.CrossRefGoogle ScholarPubMed
Stroebe, M.S., Folkman, S., Hansson, R.O. & Schut, H. (2006). The prediction of bereavement outcome: Development of an integrative risk factor framework. Social Science & Medicine (1982), 63(9), 2440–2451.CrossRefGoogle ScholarPubMed
Theut, S.K., Jordan, L., Ross, L.A. & Deutsch, S.I. (1991). Caregiver's anticipatory grief in dementia: A pilot study. International Journal of Aging & Human Development, 33(2), 113–118.CrossRefGoogle ScholarPubMed
Tomarken, A., Holland, J., Schachter, S., Vanderwerker, L., Zuckerman, E., Nelson, C., et al. (2008). Factors of complicated grief pre-death in caregivers of cancer patients. Psycho-Oncology, 17(2), 105–111.CrossRefGoogle ScholarPubMed
Twycross, R. (1996). Introducing Palliative Care. Oxford: Radcliffe Medical Press, Ltd.Google Scholar
Valdimarsdottir, U., Helgason, A., Furst, C., Adolfsson, J. & Steineck, G. (2002). The unrecognized cost of cancer patients’ unrelieved symptoms: A nationwide follow-up of their surviving partners. British Journal of Cancer, 86(10), 1540–1545.CrossRefGoogle Scholar
van der Houwen, K., Stroebe, M., Stroebe, W., Schut, H., van den Bout, J. & Wijngaards-de Meij, L. (2010). Risk factors for bereavement outcome: A multivariate approach. Death Studies, 34, 195–220.CrossRefGoogle ScholarPubMed
van Doorn, C., Kasl, S.V., Beery, L.C., Jacobs, S.C. & Prigerson, H.G. (1998). The influence of marital quality and attachment styles on traumatic grief and depressive symptoms. The Journal of Nervous and Mental Disease, 186(9), 566–573.CrossRefGoogle ScholarPubMed
Vanderwerker, L. & Prigerson, H. (2004). Social support and technological connectedness as protective factors in bereavement. Journal of Loss & Trauma - Special issue: Risk and Resiliency Following Trauma and Traumatic Loss, 9(1), 45–57.CrossRefGoogle Scholar
Vanderwerker, L.C., Jacobs, S.C., Parkes, C.M. & Prigerson, H.G. (2006). An exploration of associations between separation anxiety in childhood and complicated grief in later life. Journal of Nervous and Mental Disease, 194(2), 121–123.CrossRefGoogle ScholarPubMed
Walsh-Burke, K. (2000). Matching bereavement services to level of need. Hospice Journal, 15(1), 77–86.CrossRefGoogle ScholarPubMed
Walsh, K., King, M., Jones, L., Tookman, A. & Blizard, R. (2002). Spiritual beliefs may affect outcome of bereavement: Prospective study. BMJ: British Medical Journal, 324(7353), 1551–1551.CrossRefGoogle ScholarPubMed
Wayment, H.A. & Vierthaler, J. (2002). Attachment style and bereavement reactions. Journal of Loss & Trauma, 7(2), 129–149.CrossRefGoogle Scholar
Wijngaards-de Meij, L., Stroebe, M., Schut, H., Stroebe, W., van den Bout, J., van der Heijden, P., et al. (2005). Couples at risk following the death of their child: Predictors of grief versus depression. Journal of Consulting and Clinical Psychology, 73(4), 617–623.CrossRefGoogle ScholarPubMed
Wilson, D.C. (1988). The ultimate loss: The dying child. Loss, Grief & Care, 2(3), 125–130.Google Scholar
Zhang, B., El-Jawahri, A. & Prigerson, H.G. (2006). Update on bereavement research: Evidence-based guidelines for the diagnosis and treatment of complicated bereavement. Journal of Palliative Medicine, 9(5), 1188–1203.CrossRefGoogle ScholarPubMed
Zisook, S. & Kendler, K. (2007). Is bereavement-related depression different than non-bereavement-related depression? Psychology and Medicine, 37(6), 779–794.CrossRefGoogle ScholarPubMed
Zisook, S. & Shuchter, S. (1993). Major depression associated with widowhood. American Journal of Geriatric Psychiatry, 1(3), 316–326.CrossRefGoogle ScholarPubMed
Zisook, S. & Shuchter, S.R. (1991). Depression through the first year after the death of a spouse. American Journal of Psychiatry, 148(10), 1346–1352.Google ScholarPubMed
Zivin, K. & Christakis, N.A. (2007). The emotional toll of spousal morbidity and mortality. American Journal of Geriatric Psychiatry, 15(9), 772–779.CrossRefGoogle ScholarPubMed
Figure 0

Fig. 1. Bereavement Risk Assessment Tool (BRAT).

Figure 1

Table 1. Support from the literature of BRAT's bereavement risk and positive outcome factors

Figure 2

Table 2. Examples of default bereavement service assignment matched with BRAT risk level

Figure 3

Table 3. Participant Demographics (N = 36)

Figure 4

Table 4. Frequency of BRAT risk levels by bereaved case including expert BRAT risk levels and participants’ percent agreement with the experts

Figure 5

Table 5. Frequency of condensed* BRAT risk levels by bereaved case including the condensed expert BRAT risk levels and participants’ percent agreement with the experts