Hostname: page-component-745bb68f8f-5r2nc Total loading time: 0 Render date: 2025-02-11T02:29:19.707Z Has data issue: false hasContentIssue false

Psychometric analysis of the Melancholia Scale in trials with non-pharmacological augmentation of patients with therapy-resistant depression

Published online by Cambridge University Press:  08 October 2013

Per Bech*
Affiliation:
Psychiatric Research Unit, Psychiatric Centre North Zealand, Copenhagen University Hospital, Hillerød, Denmark
Lise Lauritzen
Affiliation:
Psychiatric Research Unit, Psychiatric Centre North Zealand, Copenhagen University Hospital, Hillerød, Denmark
Marianne Lunde
Affiliation:
Psychiatric Research Unit, Psychiatric Centre North Zealand, Copenhagen University Hospital, Hillerød, Denmark
Mogens Unden
Affiliation:
Psychiatric Research Unit, Psychiatric Centre North Zealand, Copenhagen University Hospital, Hillerød, Denmark
Lone Christina Hellström
Affiliation:
Psychiatric Research Unit, Psychiatric Centre Bispebjerg, Copenhagen University Hospital, Copenhagen, Denmark
Claudio Csillag
Affiliation:
Psychiatric Research Unit, Psychiatric Centre North Zealand, Copenhagen University Hospital, Hillerød, Denmark
Klaus Martiny
Affiliation:
Psychiatric Department O, Psychiatric Centre Copenhagen, Copenhagen University Hospital, Copenhagen, Denmark
*
Per Bech, Psychiatric Research Unit, Psychiatric Centre North Zealand, University of Copenhagen, Dyrehavevej 48, DK-3400 Hillerød, Denmark. Tel: +45 38 64 30 95; Fax: +45 48 26 38 77; E-mail: Per.bech@regionh.dk
Rights & Permissions [Opens in a new window]

Abstract

Objective

The Melancholia Scale (MES) consists of the psychic core items of the Hamilton Depression Scale (HAM-D6) (depressed mood, interests, psychic anxiety, general somatic, guilt feelings, and psychomotor retardation) and the neuropsychiatric items of the Cronholm–Ottossen Depression Scale. Patients resistant to anti-depressant medication (therapy-resistant depression) have participated in our trials with non-pharmacological augmentation. On the basis of these trials, we have evaluated to what extent the neuropsychiatric subscale of the MES (concentration difficulties, fatigability, emotional introversion, sleep problems, and decreased verbal communication) is a measure of severity of apathia when compared with the HAM-D6 subscale of the MES.

Methods

We have focused on rating sessions at baseline (week 0) and after 2 and 4 weeks of therapy in four clinical trials on therapy-resistant depression with the following augmentations: electroconvulsive therapy, bright light therapy, transcranial magnetic stimulation or pulsed electromagnetic fields, and wake therapy. The item response theory model constructed by Mokken has been used as the psychometric validation of unidimensionality. For the numerical evaluation of transferability, we have tested item ranks across the rating weeks.

Results

In the Mokken analysis, the coefficient of homogeneity was above 0.40 for both the HAM-D subscale and the apathia subscale at week 4. The numerical transferability across the weeks was statistically significant (p < 0.05) for both subscales.

Conclusion

The apathia subscale is a unidimensional scale with acceptable transferability for the measurement of treatment-resistant symptoms, analogue to the psychic core subscale (HAM-D6).

Type
Original Articles
Copyright
Copyright © Scandinavian College of Neuropsychopharmacology 2013 

Significant outcomes

  • The apathia subscale of the Melancholia Scale (MES) is psychometrically valid in the measurement of treatment-resistant depression severity.

  • When using the MES we need to evaluate the outcome on both the Hamilton Depression Scale (HAM-D6) and the apathia subscale in therapy-resistant depression.

Limitations

  • The ability of the MES apathia subscale to discriminate between the different augmentations has not been evaluated.

  • Concurrent validity with other apathia analogue scales has not been evaluated.

  • The treatment length of 4 weeks might be optimal for electroconvulsive therapy (ECT), but not for the other non-pharmacological strategies in our study.

Introduction

The lack of biological markers for depression is one of the major problems when classifying depression as a medical disorder, according to the latest revision of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5) (1). Therefore, the validity of clinical depression is while grading the depressive states into mild, moderate, or severe, essentially a symptom-based concern in which the use of item response theory models is the most appropriate validation procedure (Reference Bech2).

Item response theory models differ from principal component analysis or factor analysis by estimating an item difficulty for each symptom to evaluate to what extent the items in a depression symptom rating scale cover the dimension of depression severity from no, mild, moderate, and marked. In the non-parametric Mokken analysis (Reference Bech2,Reference Mokken3), it is the mean score of each depression item that shows the item difficulty. Thus, items with high mean scores are less difficult than items with low mean scores. According to the item response theory models, the hierarchy of item difficulties is an order by which the items with low mean scores have to be preceded by the items with higher mean scores (Reference Bech2,Reference Licht and Bech4,Reference Licht, Qvitzau, Allerup and Bech5).

In patients with therapy-resistant depression, we have identified a group of items that might be the core items maintaining these patients in their resistance to anti-depressant medication (Reference Andreasson, Liest and Lunde6). These items are part of the symptom universe on the MES (Reference Bech7) and include tiredness, concentration and/or memory difficulties, and sleep disturbances (Reference Andreasson, Liest and Lunde6). Healy (Reference Healy8) refers to this group of items as the neuropsychological features of depression, whereas he (Reference Healy8) refers to the symptoms of depressed mood, guilt feelings, or helplessness as the psychic features of depression.

On this background, we have reanalysed our previous studies with the MES in patients resistant to anti-depressant medication to evaluate, by the use of the Mokken item response theory model, those two features of depression, the neuropsychological dimension, which we will call the neuropsychiatric apathia dimension, and the psychic dimension which we consider to be the cognitive–behavioural core dimension of depression as measured by the HAM-D6, a depression subscale on the HAM-D6 (Reference Bech2).

Methods

Rating scales

MES

Table 1 shows the 11 items included in the MES (Reference Bech2,Reference Bech7). Each item has detailed answer categories from 0 = not present to 4 = present in extreme degree. The theoretical score range is therefore from 0 to 44. The standardisation of the MES total score is: 0–6 = no depression, 7–10 = doubtful depression, 11–14 = mild depression, 15–24 = moderate depression, and 25–44 = severe depression.

Table 1 The MES scales with the items included in the HAM-D6 and apathia subscales

HAM-D6, Hamilton Depression Scale; MES, Melancholia Scale.

Clinically, the MES is based on the HAM-D6, which contains the core items of depression within the HAM-D6 (Reference Bech7), as illustrated in Table 1 [depressed mood, tiredness, work and interests, psychic anxiety, guilt, and decreased motor activity (retardation)]. The other MES items are from the Cronholm–Ottosson Depression Scale (Reference Ottosson9,Reference Bech10). Table 1 shows the items on the MES included in the syndrome identified on the MES in therapy-resistant depression (Reference Andreasson, Liest and Lunde6), the apathia subscale (tiredness, concentration/memory problems, sleep disturbances, emotional introversion, and decreased verbal communication). Finally, the item of suicidal thoughts has been considered as an independent symptom.

Patients

Over the past two decades, we have used the MES as an outcome measure in our trials on therapy-resistant depression when evaluating augmentation of anti-depressant medication with non-pharmacological therapies. Thus, we have used ECT in the study by Lauritzen et al. (Reference Lauritzen, Odgaard and Clemmesen11); bright light therapy in the study by Martiny et al. (Reference Martiny, Lunde, Unden, Dam and Bech12); repetitive transcranial magnetic stimulation (rTMS) in the study by Bretlau et al. (Reference Bretlau, Lunde, Lindberg, Unden, Dissing and Bech13) and transcranial pulsed electromagnetic fields (T-PEMF) in the study by Martiny et al. (Reference Martiny, Lunde and Bech14); and finally the chronotherapeutic intervention (wake and light therapy) in the study by Martiny et al. (Reference Martiny, Refsgaard and Lund15).

From these trials, we have focussed on the MES ratings at week 0 (baseline), week 2, and week 4 during the acute augmentation, unrelated to anti-depressant medication and whether the augmentation therapy in question (bright light, rTMS, or T-PEMF) was active or inactive (sham). On this background we have the following four groups of patients:

  1. (a) A total of 67 patients from the ECT study (Reference Lauritzen, Odgaard and Clemmesen11), who were available at all ratings (week 0, 2, and 4). These patients scored 18 or more on the HAM-D17 at baseline where they all fulfilled the DSM-III-R criteria for major depression. There were 47 women and 20 men, with a mean age of (SD) 64.4 (13.5) years. The MES raters were members of the Danish University anti-depressant Group (DUAG) and had obtained adequate inter-rater reliability on the MES.

  2. (b) A total of 84 patients from the bright light study (Reference Martiny, Lunde, Unden, Dam and Bech12) were available at all ratings (week 0, 2, and 4). These patients scored 13 or more on the HAM-D17 at baseline where they fulfilled the DSM-IV criteria for major depression. There were 60 women and 24 men, with a mean age of (SD) 44.6 (15.4) years. The raters were all DUAG members and had obtained an adequate inter-rater reliability on MES.

  3. (c) A total of 84 patients were available when the rTMS (Reference Bretlau, Lunde, Lindberg, Unden, Dissing and Bech13) and the T-PEMF (Reference Martiny, Lunde and Bech14) studies were combined at all ratings (week 0, 2, and 4). These patients scored 13 or more on the HAM-D17 at baseline where they fulfilled the DMS-IV criteria for major depression. There were 53 women and 31 men, with a mean age of (SD) 53.5 (11.2) years. The raters were all DUAG members and had obtained an adequate inter-rater reliability on MES.

  4. (d) A total of 71 patients from the chronotherapeutic augmentation (wake and light therapy) were available at all ratings (week 0, 2, and 4) in the Martiny et al. (Reference Martiny, Refsgaard and Lund15) study. These patients scored 13 or more on the HAM-D17 at baseline where they fulfilled the DSM-IV criteria for major depression. There were 41 women and 30 men, with a mean age of (SD) 47.9 years (11.5). The raters were all DUAG members and had obtained an adequate inter-rater reliability on MES.

Psychometrics

Homogeneity

The Loevinger coefficient of homogeneity within the Mokken analysis (Reference Mokken3) was used as an indicator of the non-parametric item response theory analysis in accordance with the programme for polytomous items (Reference Molenaar, Debels and Sijtsna16). Because this is a coefficient, the magnitude is dependant on the score distribution. At baseline (week 0) and also at week 2, the dispersion might be too restricted, whereas at week 4 the dispersion is optimal (Reference Licht, Qvitzau, Allerup and Bech5). In our interpretation of the coefficient of homogeneity, we have therefore relied especially on the results from week 4. According to Mokken (Reference Mokken3), a coefficient between 0.30 and 0.39 is only just acceptable, whereas a coefficient of 0.40 or higher is adequate for unidimensionality.

Transferability

This is the ability of a scale to measure the same dimension across weekly ratings (Reference Bech2,Reference Bech17). For the numerical evaluation of transferability (i.e. the extent to which the rank order of the items of a scale in terms of mean score at week 0, 2, and 4 was sustained), we used the non-parametric Friedman test (Reference Siegel18). The level of statistical significance was p < 0.05.

Results

Table 2 shows the MES results from the ECT study. At week 4, the coefficient of homogeneity was equal to or higher than 0.40 for the total MES, the HAM-D6, and the apathia scale. The reduction of scores between week 0 and week 4 is rather similar for all three scales, namely, 62.5% for the whole MES, 65.6% for HAM-D6, and 60.2% for the apathia scale.

Table 2 ECT/depression (HAM-D6)/MES [Lauritzen et al. (Reference Lauritzen, Odgaard and Clemmesen11)]

ECT, electroconvulsive therapy; HAM-D6, Hamilton Depression Scale; MES, Melancholia Scale.

Table 3 shows the MES results for the bright light study. At week 4, the coefficient of homogeneity was equal to or higher for the whole MES and the HAM-D6, whereas the apathia scale obtained a coefficient of 0.38, that is, only just acceptable. The reduction of scores between week 0 and week 4 is rather similar for all three scales, namely, 39.2% for the whole MES, 37.2% for HAM-D6, and 33.9% for the apathia scale.

Table 3 MES depression (HAM-D6) [Martiny et al. (Reference Martiny, Lunde, Unden, Dam and Bech12)]

HAM-D6, Hamilton Depression Scale; MES, Melancholia Scale.

Table 4 shows the MES results from the rTMS/T-PEMF trials. At week 4, the coefficient of homogeneity was 0.40 or higher for all three scales. The reduction from week 0 to week 4 is rather similar for all three scales, namely, 35.3% for the whole MES, 33.2% for HAM-D6, and 31.4% for the apathia scales.

Table 4 MES depression (HAM-D6) (0–24) [Bretlau et al. (Reference Bretlau, Lunde, Lindberg, Unden, Dissing and Bech13), Martiny et al. (Reference Martiny, Lunde and Bech14)]

HAM-D6, Hamilton Depression Scale; MES, Melancholia Scale.

Table 5 shows the MES results from the wake-therapy trial. At week 4, the coefficient of homogeneity was above 0.40 for all three scales, and already at week 2 the coefficient of homogeneity was 0.40 or higher for all scales. The results from week 0 to week 4 are rather similar for all three scales, namely, 40.3% for the whole MES, 37.9% for the HAM-D6, and 36.0% for the apathia scale.

Table 5 MES depression (HAM-D6) (0–24) [Martiny et al. (Reference Martiny, Refsgaard and Lund15)]

HAM-D6, Hamilton Depression Scale; MES, Melancholia Scale.

Transferability for the HAM-D6, the apathia subscale, and for the full MES was obtained in all four trials. Thus, the rank order of the HAM-D6 items in Table 2 (work and interests, depressed mood, tiredness, psychic anxiety, guilt feelings, and motor inhibition) was sustained over the weeks (χ 2 = 13.9, which with 5 degrees of freedom gives p = 0.017) and the rank order of the apathia items (Table 2; concentration, emotional introversion, tiredness, decreased verbal communication) was sustained over the weeks (χ 2 = 9.9, which with 4 degrees of freedom gives p = 0.043). The rank order in the light augmentation (Table 3) of the HAM-D6 items (work and interests, depressed mood, tiredness, psychic anxiety, guilt feelings, and motor inhibition) was sustained over the weeks (χ 2 = 15.0, which with 5 degrees of freedom gives p = 0.010). The rank order of the apathia items (Table 3; concentration, emotional introversion, tiredness, decreased verbal communication) was sustained over the weeks (χ 2 = 11.5, which with 4 degrees of freedom gives p = 0.022).

The rank order in the rTMS/P-PEMF trials (Table 4) of the HAM-D6 items (work and interests, depressed mood, tiredness, psychic anxiety, guilt feelings, and motor inhibition) was sustained over the weeks (χ 2 = 15.0, which with 5 degrees of freedom gives p = 0.010). The rank order of the apathia items (Table 4; concentration, emotional introversion, tiredness, decreased verbal communication) was sustained over the weeks (χ 2 = 12.0, which with 4 degrees of freedom gives p = 0.017).

The rank order in the chronotherapeutic augmentation (Table 5) of the HAM-D6 items (work and interests, depressed mood, tiredness, psychic anxiety, guilt feelings, and motor inhibition) was sustained over the weeks (χ 2 = 13.0, which with 5 degrees of freedom gives p = 0.024). The rank order of the apathia items (Table 5; concentration, emotional introversion, tiredness, decreased verbal communication) was sustained over the weeks (χ 2 = 11.9, which with 4 degrees of freedom gives p = 0.018).

Discussion

In this analysis of non-pharmacological augmentations in patients with treatment-resistant depression, the MES total score was found to be a sufficient statistic of depression severity in itself by obtaining a coefficient of homogeneity of 0.40 after 4 weeks of therapy. This is in accordance with the results obtained by Licht and Bech (Reference Licht and Bech4).

It is a consequence of our finding that, according to the Mokken analysis, the MES is a unidimensional scale that the two subscales (the HAM-D6 and the apathia scale) also show unidimensionality. However, the numerical analysis of transferability demonstrated a slight difference in the HAM-D6 rank order of items when comparing the ECT study (Table 2) with the other studies (Tables 35). In the ECT study, the patients had to score 18 or more on the HAM-D17 at baseline and the mean scores for the inpatient were ∼30 on the HAM-D17 (Reference Lauritzen, Odgaard and Clemmesen11). In the other trials with outpatients, the patients had to score 13 or more. The HAM-D17 and the mean HAM-D17 scores in these trials were ∼22. A depressed patient before ECT seems to follow the ‘King Lear Principle’ (‘…when the greater malady is fixed, the lesser is scarce felt…’) in considering the symptom of tiredness less inclusive than depressed mood and work and interests (Reference Foulds19).

The occurrence of the hierarchical pattern in the incidence of items by their ranked mean scores is in the Mokken analysis tested week for week and not dynamically across weeks. By use of the Friedman two-way analysis of variance by ranks, we have been able to show that the ranked mean scores were independent of the rating occurrences (weeks). Thereby, we have demonstrated an adequate level of transferability both of the total MES scale but also of the two subscales of psychic depression (HAM-D6) and apathia.

In depressed outpatients, the symptom of tiredness is the most prevalent compared with depressed mood and psychic anxiety (Reference Healy8,Reference Maj20). In recurrent brief depression, tiredness or lack of energy is a most dominating symptom, and in recurrent states of hypomania increased energy is a most dominating symptom (Reference Angst21). According to Healy (Reference Healy and Williams22), the patients often find it difficult to evaluate to what extent ‘fatigability’ is a physical or a mental manifestation. In the context of HAM-D6, the item of tiredness is typically scored as a physical symptom in agreement with Hamilton (Reference Hamilton23). In the context of the neuropsychiatric apathia scale, the item of tiredness is typically scored as a more mental symptom such as lassitude (Reference Andreasson, Liest and Lunde6). In the original HAM-D6, the item of general somatics (fatigability) is scored on a Likert scale from 0 to 2, in contrast with the other four items, which are scored on a Likert scale from 0 to 4. In the MES-derived HAM-D6, the item of fatigability is scored from 0 to 4. Therefore, when using the mean item score to indicate item difficulty within the Mokken analysis, the item of fatigability is most inclusive (highest mean score), whereas in the original score this item of fatigability is less inclusive (Reference Bech2).

The numerical evaluation of transferability of the two subscales in the MES (namely, the HAM-D6 and the apathia scale) in terms of what was invariant across the weeks of assessment as to the rank order of items reached a clear statistical significance. This invariance is the essential matter in the measurement of depressive states (Reference Bech2). The validity of the neuropsychiatric apathia syndrome to predict the ability of depressed patients to restore social functioning in terms of return to their work after a depressive episode has been found to be of statistical significance (Reference Hellström, Bech, Nordentoft, Hansen and Eplov24).

Acknowledgements

The authors thank Professor of Theoretical Statistics Peter Allerup for his advice.

Financial Support

None.

Statement of Interest

Per Bech, Lise Lauritzen, Marianne Lunde, Mogens Unden and Lone Christina Hellström have no statement of interest to declare. Claudio Csillag has received a travel grant from Servier. Klaus Martiny has occasionally served as a speaker for pharmaceutical companies with an interest in the drug treatment of affective disorders (Servier and Eli Lilly).

Authors’ Contributions

Per Bech: conception, design, analysis, interpretation of data, and draught of article. Lise Lauritzen, Marianne Lunde, Mogens Unden: acquisition of data. Lone Christina Hellström: conception, design, and draught of article. Claudio Csillag: draught of article and critical revision. Klaus Martiny: acquisition of data, critical revision.

References

1.American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), 5th edn.Washington, DC: American Psychiatric Association, 2013.Google Scholar
2.Bech, P. Clinical Psychometrics. Oxford: Wiley Blackwell, 2012.CrossRefGoogle Scholar
3.Mokken, RJ. Theory and Practice of Scale Analysis. Berlin: Mouton, 1971.Google Scholar
4.Licht, RW, Bech, P. ‘The Hamilton Depression Rating Scale: Has the gold standard become a lead weight?’: comment. Am J Psychiatry Dec 2005;162:23942395.Google Scholar
5.Licht, RW, Qvitzau, S, Allerup, P, Bech, P. Validation of the Bech-Rafaelsen Melancholia Scale and the Hamilton Depression Scale in patients with major depression; is the total score a valid measure of illness severity? Acta Psychiatr Scand 2005;111:144149.Google Scholar
6.Andreasson, K, Liest, V, Lunde, Met al. Identifying patients with therapy-resistant depression by using factor analysis. Pharmacopsychiatry 2010;43:252256.Google Scholar
7.Bech, P. The Bech-Rafaelsen Melancholia Scale (MES) in clinical trials of therapies in depressive disorders: a 20-year review of its use as outcome measure. Acta Psychiatr Scand 2002;106:252264.Google Scholar
8.Healy, D. The Suspended Revolution: Psychiatry and Psychology Re-Examined. New York: Faber & Faber, 1989.Google Scholar
9.Ottosson, JO. Experimental studies of the mode of action of electroconvulsive therapy: introduction. Acta Psychiatr Scand Suppl 1960;35:56.Google Scholar
10.Bech, P. The Cronholm-Ottosson Depression Scale: the first depression scale designed to rate changes during treatment. Acta Psychiatr Scand 1991;84:439445.Google Scholar
11.Lauritzen, L, Odgaard, K, Clemmesen, Let al. Relapse prevention by means of paroxetine in ECT-treated patients with major depression: a comparison with imipramine and placebo in medium-term continuation therapy. Acta Psychiatr Scand 1996;94:241251.Google Scholar
12.Martiny, K, Lunde, M, Unden, M, Dam, H, Bech, P. Adjunctive bright light in non-seasonal major depression: results from patient-reported symptom and well-being scales. Acta Psychiatr Scand 2005;111:453459.Google Scholar
13.Bretlau, LG, Lunde, M, Lindberg, L, Unden, M, Dissing, S, Bech, P. Repetitive transcranial magnetic stimulation (rTMS) in combination with escitalopram in patients with treatment-resistant major depression: a double-blind, randomised, sham-controlled trial. Pharmacopsychiatry 2008;41:4147.Google Scholar
14.Martiny, K, Lunde, M, Bech, P. Transcranial low voltage pulsed electromagnetic fields in patients with treatment-resistant depression. Biol Psychiatry 2010;68:163169.CrossRefGoogle ScholarPubMed
15.Martiny, K, Refsgaard, E, Lund, Vet al. A 9-week randomized trial comparing a chronotherapeutic intervention (wake and light therapy) to exercise in major depressive disorder patients treated with duloxetine. J Clin Psychiatry 2012;73:12341242.Google Scholar
16.Molenaar, IW, Debels, P, Sijtsna, K. User's Manual MSP, a Program for Mokken Scale Analyses for Polytomous Items (Version 3.0). Groeningen, the Netherlands: ProGAMMA, 1994.Google Scholar
17.Bech, P. Rating scales for affective disorders: their validity and consistency. Acta Psychiatr Scand Suppl 1981;295:1101.Google Scholar
18.Siegel, S. Nonparametric Statistics for the Behavioural Sciences. New York: McGraw Hill, 1956.Google Scholar
19.Foulds, GA. The Hierarchical Nature of Personal Illness. London: Academic Press, 1976.Google Scholar
20.Maj, M. Development and validation of the current concept of major depression. Psychopathology 2012;45:135146.Google Scholar
21.Angst, J. Treated versus untreated major depressive episodes. Psychopathology 1998;31:3744.Google Scholar
22.Healy, D, Williams, JM. Dysrhythmia, dysphoria, and depression: the interaction of learned helplessness and circadian dysrhythmia in the pathogenesis of depression. Psychol Bull 1988;103:163178.CrossRefGoogle ScholarPubMed
23.Hamilton, M. Development of a rating scale for primary depressive illness. Br J Soc Clin Psychol 1967;6:278296.CrossRefGoogle ScholarPubMed
24.Hellström, LC, Bech, P, Nordentoft, M, Hansen, JL, Eplov, LF. Sherpa. The effect of an early supported employment intervention for patients with anxiety or an affective disorder – a randomised clinical trial. Trials 2013, in press.Google Scholar
Figure 0

Table 1 The MES scales with the items included in the HAM-D6 and apathia subscales

Figure 1

Table 2 ECT/depression (HAM-D6)/MES [Lauritzen et al. (11)]

Figure 2

Table 3 MES depression (HAM-D6) [Martiny et al. (12)]

Figure 3

Table 4 MES depression (HAM-D6) (0–24) [Bretlau et al. (13), Martiny et al. (14)]

Figure 4

Table 5 MES depression (HAM-D6) (0–24) [Martiny et al. (15)]