Introduction
Determining how CBT is delivered is an important aspect of practice. For example, researchers need to identify fidelity to the treatment model in a clinical trial, training courses need to appraise the acquisition of skills in their students, and supervisors need some way of monitoring the development and maintenance of competence. To achieve this we need structured observation scales. To date, a limited number of instruments have been developed, of which two are commonly applied. The first is the Cognitive Therapy Rating Scale (CTRS) in its original (Young and Beck, Reference Young and Beck1988) and revised forms (e.g. the Cognitive Therapy Scale-Revised (CTS-R): Blackburn et al., Reference Blackburn, James, Milne, Baker, Standart and Garland2001); the second is the CBT subscale of the Collaborative Study Psychotherapy Rating Scale (CSPRS: Hollon et al., Reference Hollon, Evans, Auerbach, DeRubeis, Elkin and Lowery1988). While use of the latter scale seems to be largely restricted to research trials, the CTRS/CTS-R is widely employed by researchers, trainers and clinicians.
This paper introduces a new scale for structured observation of CBTFootnote 1 . Although the development of a new scale is not dependent on deficiencies in prior ones, the CTRS/CTS-R is problematic in at least three areas. First, although groups of raters working together can achieve high levels of concordance, the reliability of ratings by independent judges is uncertain (e.g. Reichelt, James and Blackburn, Reference Reichelt, James and Blackburn2003). Second, the scale subsumes the use of change techniques under one item, although this is an area of competence where there is much to be said for specificity. Third, since the inception of the scale it has become clear that CBT needs to be adapted when working with different conditions (particularly the anxiety disorders), making the generic nature of the CTS increasingly problematicFootnote 2 .
Method
The UCL scale for Structured Observation is rooted in the competence framework for CBT (Roth and Pilling, Reference Roth and Pilling2007), developed as part of the English Improving Access to Psychological Therapy (IAPT) programme, and used to generate the IAPT CBT curriculum for people presenting with anxiety and depression. The framework organizes the delivery of CBT into discrete areas of activity, and identifies the knowledge and skills that underpin all variants of CBT as well as specific CBT skills that are applied when working with specific conditions or presentations.
The framework also includes a domain of Generic Therapeutic Competences, areas of knowledge and skills that are common across therapy modalities (for example, relational competences, such as alliance building and repair) and skills associated with the management of sessions (for example, using measures, responding to emotional expression, or ending sessions). Although generic competences are necessary skills for the effective delivery of therapy, it is helpful to separate them from CBT-specific skills. As such two parallel scales were developed, both of which would usually be administered, focusing on generic and CBT competences respectively.
Scale development
The CBT competence framework includes a “map” setting out the discrete areas of clinical activity that constitute CBT practice, and this was used to identify a set of items for the scale. This initial set was modified on the basis of feedback from clinicians piloting early versions of the scale; the resulting set of 26 items is organized into four sections:
-
1. Underpinning CBT techniques.
-
2. Change techniques based on discussion and experiential methods.
-
3. Change techniques deployed for specific conditions.
-
4. Reviewing the session as whole.
A similar procedure was used to develop a 13-item Core and Generic Therapeutic Competences scale.
Each item is anchored with synoptic behaviourally-specific descriptions that indicate what an observer would expect to see if the competence were being performed appropriately (with these descriptions drawn from those contained in the framework). The descriptors are set at the level that an expert practitioner would demonstrate, and to which a trainee therapist would aspire. By way of illustration, Table 1 shows the behavioural anchors for two items: “agenda setting and structuring sessions” (one of the underpinning CBT techniques), and “implementing CBT using a collaborative approach” (found in the section “reviewing the session as a whole”). These anchors can include multiple concepts, reflecting the fact that the activity they describe is exemplified by a set of actions that (even if different in their focus) are coherent in their intent: all need to be present for the item to be rated as competently delivered. For example, however well an agenda is set, the session needs to be structured in a way that enables this agenda to be realised; one activity without the other is unhelpful.
Table 1. Examples of scale items and their behavioural anchors
Ratings are made on a 5-point Likert scale, with each point linked to anchors that specify both the extent to which a skill is present and the degree to which it needs to be developed (ranging from “Competence not demonstrated or requires major development” through to “Competence demonstrated very well and requires no substantive development”). A short guide to the generic and CBT scales sets out their rationale and the way in which they are rated and scored, and directs raters to the CBT framework for a fuller description of the competences.
Most competence scales require raters to score all the items, but this can be problematic if areas of activity are not relevant to a specific intervention package or to a specific session. This observation is particularly pertinent to the present scale because of the detail with which it specifies change mechanisms. As such, raters have the option of marking an area of activity as “not-applicable” when it is absent from the session, but its absence is expected or is justified by the context. For example, “working with imagery” could be rated “non-applicable” if imagery did not feature in a session, but if a client described obviously relevant imagery that the therapist did not address, then the rater would score this as an area needing development. Clearly judgment is needed to decide whether – in the context of the session - an area of activity is appropriately absent, or whether its absence should flag a concern.
Results
Feedback on the use of the scales
Detailed feedback on the scales has been obtained through several routes: iterative review from senior CBT practitioners who commented on the scales at different points in their development, piloting of the generic and CBT scales by trainee clinical psychologists and their supervisors, feedback from raters conducting a pilot study of the scale's reliability, and feedback from raters in the preparatory stages of an appropriately-powered inter-rater reliability study. This process of iterative feedback has been used to make improvements to both the scale and the rating system.
Conclusion
Psychometric properties of the scale and inter-rater reliability
Understanding the psychometric properties of the scale is the next stage in its development, with inter-rater reliability a primary and critical consideration if it is to be used to make meaningful summative judgments. This is especially important in relation to setting a cut-point for competence, which at present is represented qualitatively (in that a score of 3 or more indicates that the competence is demonstrated, but to varying degrees of proficiency). An ongoing study is exploring these issues, with a fully-crossed design in which six independent and experienced judges rate the same 30 recordings on both the UCL scale and the CTS-R. This should afford a detailed picture of the properties of the scale and yield information on the performance of individual items within it.
Acknowledgements
Conflict of interest: The author has no conflicts of interest with respect to this publication.
Supplementary Material
An extended version is also available online under the Brief Clinical Report Supplementary Materials tab in the table of contents. Please visit http://dx.doi.org/10.1017/S1352465816000011
Comments
No Comments have been published for this article.