Introduction
Cognitive therapy is well established as an effective treatment for a wide range of disorders, and is widely practised by a range of mental health professionals (Roth and Fonagy, Reference Roth and Fonagy2004). Evidence for the effectiveness of cognitive therapy has come from controlled clinical trials. In clinical trials, the competence with which therapy is administered is ensured through rigorous training of therapists, close supervision, and independent assessments of competence. By contrast, in routine clinical practice, therapists may have less rigorous training, sporadic supervision, and little scrutiny of their competence in cognitive therapy. It cannot therefore be assumed that cognitive therapy in routine clinical situations is delivered to the same standards of competence that have been shown to be effective in trials.
In the UK, considerable resources are being invested in plans to increase access to psychological therapies, particularly cognitive behavioural therapies, through the Improving Access to Psychological Therapies (IAPT) programme (Department of Health, 2007). A substantial expansion of therapy provision is anticipated. Within such expanded services, it is essential that attention is given to ensuring that the therapy delivered is delivered competently. Information on the factors that might be useful in ensuring quality control of therapy provision will be valuable.
In the routine practice of cognitive therapy, there is little evidence that models of professional regulation or accreditation ensure therapeutic competence (e.g. Brosan, Reynolds and Moore, Reference Brosan, Reynolds and Moore2007). Individual therapists bear a large part of the responsibility for quality control. This quality control depends in turn on the ability of individual cognitive therapists to evaluate their competence accurately. If therapists are aware of their shortcomings, then they are likely to be able to address these and to improve. If they are not aware of their weaknesses, then suboptimal practice is likely to continue.
The potential inaccuracy of people's self-assessments of their performance has been documented in many fields (see Dunning, Reference Dunning2006), including psychotherapy (e.g. Chevron and Rounsaville, Reference Chevron and Rounsaville1983). Several studies have compared therapists’ self-evaluations with evaluations by independent observers. Buckley, Conte, Plutchik and Karasu (Reference Buckley, Conte, Plutchik and Karasu1981) compared psychiatric residents’ self-ratings of their psychotherapy skills with ratings of their skills made by their supervisors. Supervisors rated the skills of less competent trainees as poorer than those of more competent trainees, whereas the self-evaluations of the two groups showed little difference. Less competent trainees were less accurate in their self-evaluation than the more competent trainees. Lafferty, Beutler and Crago (Reference Lafferty, Beutler and Crago1989) looked at therapist characteristics associated with outcome, and found that less effective therapists showed poorer self-evaluation skills, rating their patients as more involved in therapy, and as making more progress in treatment, than did an observer. In a study of Motivational Interviewing, Miller, Yahne, Moyers, Martinez and Pirritano (Reference Miller, Yahne, Moyers, Martinez and Pirritano2004) found that therapists’ self-reports of their skilfulness were unrelated to observer ratings of proficiency in practice.
These findings suggest that therapists’ evaluations of their competence may not be accurate. Despite these findings, there has been little research on the accuracy of therapists’ self-evaluations in cognitive therapy. We sought to extend the above findings in order to determine the accuracy with which therapists are able to evaluate their competence in cognitive therapy, and to investigate whether more and less competent therapists differ in the accuracy of this self-evaluation. We hypothesized that self-evaluations of skilled therapists would tend to agree with the judgements of an independent expert, whereas less competent therapists would be less accurate in their self-evaluations due to over-rating their competence.
Method
Participants
Subjects for the study were therapists who view themselves as practising cognitive therapy in their routine clinical practice. Participants were recruited by writing to: i) clinical supervisors of a clinical psychology training program; ii) people who had completed a well-known specialist cognitive therapy course; and iii) members of a national cognitive therapy organization, the British Association of Behavioural and Cognitive Psychotherapy. Forty-seven therapists agreed to take part and were sent a study pack, which included questionnaires about the therapist and client, a 120-minute blank audio cassette, the Cognitive Therapy Scale and an information sheet and consent form for clients. Participants were asked to complete the questionnaires, tape a session from the middle of therapy and return the questionnaires, rating form, and tape to the author.
Twenty-two therapists returned the tape, rating form and questionnaires (16 female, 6 male; mean age 38 years (SD = 6.5)). Ten participants were psychologists, 10 were RMN nurses, and 2 were from other mental health professions. Twelve of the therapists had post-qualification training in cognitive therapy to certificate or diploma level.
Measures
Cognitive Therapy Scale (CTS)
The Cognitive Therapy Scale was devised by Young and Beck (Reference Young and Beck1980) for rating competence in cognitive therapy from tapes of therapy sessions. It has been widely used in cognitive therapy research and has acceptable psychometric properties (Dobson, Shaw and Vallis, Reference Dobson, Shaw and Vallis1985). The current version of the CTS, adapted by Freeman, Pretzer, Fleming and Simon (Reference Freeman, Pretzer, Fleming and Simon1990), includes 13 items scored from 0 to 6, with concise descriptions of clinical activity given as anchors for points 0, 2, 4, and 6. The 13 items are divided between 3 subscales: General Interview Procedures (4 items covering agenda setting, eliciting feedback, collaboration and pacing); Interpersonal Effectiveness (3 items covering empathic skills, interpersonal effectiveness and professionalism) and Specific Cognitive-Behavioural Techniques (6 items covering guided discovery, conceptualization, focus on key cognitions, application of cognitive techniques, application of behavioural techniques and homework).
Independent rater
All therapy tapes were rated on the CTS by the therapists themselves and by an independent expert rater. This was a cognitive therapist with 8 years of cognitive therapy experience since completing a Diploma from a well-established cognitive therapy training course, who had extensive experience of using the CTS for supervisory purposes. To establish the reliability of the independent ratings, a second expert rater (who was a therapist with 10 years’ experience since formal cognitive therapy training) rated five randomly selected tapes. The inter-rater reliability was high (Pearson's r = .82; p < .05).
Procedure
Therapists taped a mid-treatment cognitive therapy session and rated this using the CTS. Therapists also provided information about their profession and training. The tapes were subsequently rated by the expert rater, who was blind to all information about the therapist.
Ethics
Ethical approval was given by the local NHS research ethics committee. Information about the study was provided to all potential participants. Therapists provided information sheets to the patients to whom taping was suggested and obtained informed consent for the tape to be used for research purposes. Patients participated anonymously, whereas therapists could elect to provide contact details in order to receive the ratings from their tape.
Results
Agreement between subjects’ own ratings of their therapeutic competence in the taped session and the ratings of the expert rater was assessed by correlations between the sets of ratings. The Spearman correlation between self- and expert ratings on the CTS across all therapists was 0.57 (n = 22, p < .01), reflecting a modest but significant degree of agreement.
The magnitude of the actual differences between self- and expert ratings was then examined. The mean of self-rated scores on the CTS was 47 (range 28–65), whereas the mean of scores from the expert rater on the CTS was 39 (range 19–62). Therapists’ self-ratings were significantly higher than the ratings from the expert rater (t = 3.3, df = 21, p < .01).
Therapists were then split into “Competent Therapists” and “Less Competent Therapists” on the basis of the expert rater's CTS ratings. A score of 39 on the CTS was used as a cut-off, corresponding to the cut-off score for competence used in the NIMH Collaborative Depression Study (Shaw et al., Reference Shaw, Elkin, Yamaguchi, Olmsted, Vallis, Dobson, Lowery, Sotsky, Watkins and Imber1999). Twelve therapists scored 39 or above (Competent Therapists) and ten scored less than 39 (Less Competent Therapists).
For each of these groups, difference scores were calculated between self and expert ratings and the means of these difference scores for the two groups are shown in Table 1. The difference between self- and expert ratings was significantly greater for less competent therapists than for competent therapists, suggesting that less competent therapists showed a significantly greater tendency to overestimate their competence than competent therapists.
Table 1. Mean differences (and standard deviations) between self-and expert ratings on the Cognitive Therapy Scale and subscales for competent and less competent therapists
To explore these results further, in particular to examine whether the overestimation of competence relates to particular aspects of competence, a similar analysis was performed on each of the separate subscales of the CTS. The differences between self- and expert ratings on each subscale for the two groups are also presented in Table 1. In this post-hoc analysis, less competent therapists over-rated their competence to a significantly greater degree than competent therapists on CTS Subscale 2 (Interpersonal Effectiveness) and CTS Subscale 3 (Cognitive-Behavioural Techniques). The difference between groups on CTS Subscale 1 (General Interview Procedures) was not significant.
Discussion
The correlational data from this study suggest moderate agreement between self-ratings of competence by cognitive therapists and ratings by an independent expert rater. However, examination of actual ratings reveals that therapists overall tend to overestimate their competence relative to expert judgements. This over-evaluation of competence was significantly greater in less competent therapists. These findings that less competent cognitive therapists do not evaluate their performance accurately are consistent with the findings from other psychotherapeutic fields.
Further post-hoc analysis of the data found that less competent therapists overestimated their competence more than competent therapists on the subscales of the CTS measuring Interpersonal Effectiveness and Cognitive and Behavioural Techniques, but not on the subscale measuring General Interview Procedures. Although no specific predictions had been made, it might be expected that less competent therapists would have greatest difficulty in evaluating their performance on items that are most specific to cognitive therapy. Consistent with this, the overestimation by less competent therapists was indeed most marked on the Cognitive and Behavioural Techniques subscale. However, the Interpersonal Effectiveness subscale contains items (e.g. empathy, professionalism) that are least specific to CBT yet also revealed significant overestimation by less competent therapists. In contrast, the items of the General Interview Procedures subscale are somewhat specific to CBT (e.g. agenda setting, eliciting feedback) and revealed no differences in overestimation of competence between competent and less competent therapists.
A further factor that may account for the differences between subscales is the basis on which ratings are made. Ratings for items on the General Interview Procedures subscale are based on criteria that are more objective (e.g. rating whether an agenda was set and followed) than ratings for the other subscales, which require more judgement (e.g. of the extent to which the therapist understood the patient's internal reality) and evaluation (e.g. of the degree of skill with which cognitive and behavioural techniques are applied). It is possible that the nature of the ratings on the General Interview Procedures subscale affords less scope for greater overrating by less competent therapists.
Explanations for the finding that less competent therapists over-evaluate their performance must include the internal processes through which self-evaluations are arrived at and external influences on these. In terms of the cognitive processes involved, less competent therapists’ judgements of competence may be influenced by less relevant cues, such as the patient's apparent degree of gratitude. By contrast, the judgements of more competent therapists may be based more firmly on relevant criteria, such as having used guided discovery to identify distressing thoughts and feelings.
External influences, particularly that of training, on this process of self-evaluation may be important. Data described in a previous report on this study showed that competence in cognitive therapy was significantly greater in those who had undergone formal post-qualification training (Brosan et al., Reference Brosan, Reynolds and Moore2007). Therapists judged as competent in this study will previously have had greater exposure through such training to independent evaluations of their own performance. This may have helped them to align their self-judgements with external feedback. Through such training, more competent therapists may also have had a greater degree of prior exposure to the CTS itself, enabling them to use it to rate their performance more accurately. An attempt was made to disentangle the effects of competence on overestimation from those of training in these data. The extent of overestimation in those therapists who had formal cognitive therapy training but were judged “less competent” (n = 3) was compared with that in those with no formal training but who were judged “competent” (n = 4). Both groups exhibited a similar degree of overestimation compared to the independent rater, but numbers were too small to be meaningful. Further research would be helpful in examining the effects of training on self-evaluations of competence.
Two methodological caveats apply to the present findings. First, the reliability of the CTS has been found to be acceptable but not high (Dobson et al., Reference Dobson, Shaw and Vallis1985). It is likely that any unreliability of the measure would weigh against producing consistent findings such as those relating to the overestimation of competence described here. A revised version of the CTS is now available that has also been validated as a training measure (Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart, Garland and Reichelt2001), which would be useful to compare with the measure used here. Second, the sample size was relatively small. Therefore, further research would be helpful to replicate the current results using the revised CTS and enough therapists who have been through formal training but not yet reached criteria for competence in order to disentangle whether overestimation relates more closely to being less trained or to being less competent.
The current findings have potentially serious implications for the practice of cognitive therapy in ordinary clinical settings. They suggest that therapists’ self-assessments of their performance are not reliable and would not satisfactorily ensure quality control of therapy in clinical settings. Furthermore, relatively less competent therapists may persist in clinical practice with little insight into the need to improve their performance. In any rapid expansion of therapy services, such as the one currently envisaged in the UK, there is a clear danger in placing excessive reliance on therapists’ own judgements in determining the quality of therapy services provided. Supervision based on therapists’ own accounts of progress in therapy, rather than direct or taped observation, may not address this problem. Therapists may require specific, expert feedback to help them reflect accurately on their performance and to make any necessary improvements. Otherwise, the expansion of therapy provision, although it may appear to be evidence-based, may propagate disturbing differences between the research trials on which cognitive therapy's reputation rests and the reality of services delivered to patients.
Acknowledgements
This study was undertaken as part of the first author's doctorate from the University of East Anglia. We are indebted to David Allison for rating the therapy tapes, and to the patients of the therapists who allowed their clinical sessions to be sent to a stranger. Most of all we are indebted to the therapists who took part.
Comments
No Comments have been published for this article.