Introduction
Monitoring treatment integrity, which ensures that a treatment was implemented as intended, is often missing from treatment studies (Moncher and Prinz, Reference Moncher and Prinz1991; Waltz et al., Reference Waltz, Addis, Koerner and Jacobson1993; Perepletchikova et al., Reference Perepletchikova, Treat and Kazdin2007). In addition to securing internal validity, treatment integrity is important as a potential mediator for evaluating the effectiveness of a specific treatment (Weck et al., Reference Weck, Bohn, Ginzburg and Stangier2011a). Thus such monitoring may facilitate identifying the most important therapeutic skills for a specific treatment and integrating them into therapist training and education (Perepletchikova and Kazdin, Reference Perepletchikova and Kazdin2005). Similar to therapeutic competence and treatment differentiation, therapeutic adherence is an important part of treatment integrity. Therapeutic adherence is defined as the extent to which a therapist applies interventions as they are described in the treatment manual (Waltz et al., Reference Waltz, Addis, Koerner and Jacobson1993). Generally, separate assessment of therapeutic adherence and competence is recommended to help distinguish between the two constructs, which often are confused (Dobson and Singer, Reference Dobson and Singer2005; Barber et al., Reference Barber, Triffleman and Marmar2007).
Four different types of therapist behaviour should be included in adherence measurement: (1) ‘therapist behaviours that are unique to that treatment modality and essential to it’ (e.g. behavioural experiments in the treatment of social phobia); (2) ‘behaviours that are essential to the treatment but not unique to it’ (e.g. cognitive techniques); (3) ‘behaviours that are compatible with the specified modality, and therefore not prohibited, but neither necessary nor unique’ (e.g. chatting at the beginning of a session); and (4) ‘behaviours that are proscribed’ (e.g. exposure elements in an exclusively cognitive treatment) (Waltz et al., Reference Waltz, Addis, Koerner and Jacobson1993; pp. 624–625).
In addition to providing a treatment manual, adequate training of the therapists and regular supervision, assessment by independent observers is recommended to ensure treatment adherence (Weck et al., Reference Weck, Bohn, Ginzburg and Stangier2011a). A common method for assessing adherence is to rate video or audio recordings, in which different treatment elements (items) are specified for each session (e.g. Resick et al., Reference Resick, Galovski, Uhlmansiek, Scher, Clum and Young-Xu2008; Gutermann et al., Reference Gutermann, Schreiber, Matulis, Stangier, Rosner and Steil2015). The rater assesses whether these specified interventions have occurred (e.g. did the therapist deliver the designated content of session one: yes or no?). Other studies have focused on the assessment of more global therapeutic adherence independent of the content of the specific treatment session. Thus for every session, the same items that reflect more global adherence are rated, such as ‘time management’, ‘agenda’ or ‘addressing problematic behaviour’. This method is particularly useful for therapies with flexible, modular interventions tailored to the individual patient, and it allows for the rating of randomly selected videos, which is less time consuming (e.g. Barber et al., Reference Barber, Triffleman and Marmar2007).
However, both methods require highly qualified raters, which imply high costs. Therefore, despite the importance of assessing treatment integrity, many studies fail to do so. When adherence is assessed, adequate measures are often missing, or the respective scales are not tested for their psychometric properties. Generally, few studies on post-traumatic stress disorder (PTSD) analyse therapeutic adherence (Barber et al., Reference Barber, Triffleman and Marmar2007). For one of the most effective and frequently evaluated PTSD treatments, cognitive processing therapy (CPT; Resick et al., Reference Resick, Monson and Chard2010), therapeutic adherence is often measured and described. Nevertheless, the psychometric properties of the instruments used have not yet been reported (e.g. Resick et al., Reference Resick, Galovski, Uhlmansiek, Scher, Clum and Young-Xu2008).
In CPT (Resick et al., Reference Resick, Galovski, Uhlmansiek, Scher, Clum and Young-Xu2008), the therapist first aims to identify and change dysfunctional beliefs related to the trauma, such as self-blame (e.g. ‘It was my fault that he raped me’). Patients gradually learn to challenge their beliefs through Socratic questioning and the use of worksheets, with the goal of generating more balanced statements and reducing negative emotions.
Patients with PTSD after childhood sexual/physical abuse (CSA/CPA) typically exhibit severe psychopathology and several co-morbid diagnoses (Green et al., Reference Green, McLaughlin, Berglund, Gruber, Sampson, Zaslavsky and Kessler2010). This fact is often neglected in treatments (e.g. Steil et al., Reference Steil, Dyer, Priebe, Kleindienst and Bohus2011). Accordingly, we modified the original CPT protocol by adding sessions, thus providing additional time within the CPT protocol to address specific co-morbid symptoms of patients with complex trauma histories. For this adaptation, we developed the Adherence Rating Scale for CPT (ARS-CPT), which solely assessed therapeutic adherence by rating video recordings of therapy sessions. This novel adherence scale combines items that refer to global therapeutic adherence with items that assess aspects of session-specific adherence. This study tested the newly developed scale to determine its psychometric properties by examining its inter-rater reliability and internal consistency. We also tested whether our items represent all facets of the CPT using ratings by independent experts. Finally, we determined whether the ARS-CPT is a user-friendly, efficient rating instrument.
Method
Treatment
The data for this study were obtained from a clinical multicentre randomized controlled trial (RCT) that compared the use of CPT with Dialectical Behaviour Therapy to treat PTSD in patients with PTSD and borderline personality symptoms (German Clinical Trials Registration ID: DRKS00006095). The inclusion criteria for the RCT were PTSD after CSA/CPA according to the structured clinical interview of DSM-IV (SKID I; Wittchen et al., Reference Wittchen, Zaudig and Fydrich1997) plus at least three criteria for borderline personality disorder (BPD). Each treatment consisted of 45 sessions within a 1-year period (for additional details, see www.traumatherapie-verbund.de). The original CPT was adapted as follows: four sessions focused on establishing rapport and developing emergency plans are followed by the 12 original CPT sessions. Subsequently, individual topics related to the trauma are addressed in 29 additional sessions.
Adherence rating scale for CPT
We determined to develop a new rating scale for assessing therapeutic adherence based on the adapted CPT treatment manual and orientated on the original CPT Therapists’ Adherence Protocol – Revised (Mcdonald et al., Reference Mcdonald, Wiltsey-Stirman, Wachen and Resick2014). We included parts (the adherence ratings) of the original CPT Therapists’ Adherence Protocol – Revised (items 4–6; for a detailed description, see Resick et al., Reference Resick, Galovski, Uhlmansiek, Scher, Clum and Young-Xu2008), which is a session-specific checklist that assesses whether interventions occurred for ‘unique and essential elements specific to each session, essential but not unique elements, acceptable but not necessary elements, and proscribed elements’ (Resick et al., Reference Resick, Galovski, Uhlmansiek, Scher, Clum and Young-Xu2008, p. 249) in each of the 12 original CPT sessions. We also developed a corresponding checklist applicable for the 29 additional more flexible sessions in our manual.
Additionally, together with a small group of experts in CPT treatment, we developed six new items (items 1–3, 7–10) that, in contrast to the original CPT Therapists’ Adherence Protocol, refer to more global therapeutic adherence and are applicable to each treatment session, which enables random selection of the treatment sessions to be rated. These items are essential for our adapted CPT protocol, which includes more flexible sessions addressing individual topics related to the trauma. The items are rated using a 3-point Likert scale with 0 (not adherent), 1 (partly implemented/adherent to some extent) and 2 (adherent), with descriptions of adherent and non-adherent behaviour based on the manual for our adapted CPT protocol.
Item 10 assesses global adherence in the therapy session using a 7-point Likert scale from 0 (not adherent) to 6 (very adherent).
Applying our new scale, we combined both approaches as described to assess adherence specified for each treatment session while considering unique elements for each session (orientated on the original CPT Therapists’ Adherence Protocol – Revised) and global adherence (newly developed items). We thereby developed a rating scale that is flexible, simple to use and applicable to each of the 41 treatment sessions (except the first four sessions). Thus our scale enables a broader application to CPT variations that differ in length and flexibility from the original protocol.
Content validation
Three CPT experts were asked to evaluate the relevance and appropriateness of the ARS-CPT items using a 4-point Likert scale from 0 (not at all relevant/appropriate) to 3 (extremely relevant/appropriate). Although the experts were not involved in the video ratings, two of them worked as supervisors in the treatment study.
Raters
The two raters, with 3 and 8.5 years of clinical experiences, respectively, had received intensive training (36 h) in CPT by its developer P. Resick. Both raters had treated patients with CPT under supervision and were involved in the development of the new adherence scale; they did not participate in the supervision of the therapists during the study. Three pilot cases were evaluated and discrepancies discussed before starting the assessments for this study. To avoid drifting apart during the study, the ratings of every fifth video were compared and discussed by the raters (but not changed afterwards).
Participants and therapists
The study examined 30 videotapes of seven therapists and eight patients: two to three of 41 videotapes were randomly selected for each patient, including one of the 12 CPT core sessions and at least one from the subsequent additional sessions.
The patients were all women with a mean age of 31.13 years (SD = 7.7; range 22–43). The PTSD diagnosis was related to CPA in two cases and to CSA in six cases. All of the patients met at least three criteria for BPD. They also had been diagnosed with two to five co-morbid mental disorders, typically major depression and/or anxiety disorders according to the SKID I interview (Wittchen et al., Reference Wittchen, Zaudig and Fydrich1997). All of the therapists were clinical psychologists who had been trained in CPT by the developer, P. Resick, in a 5-day workshop and were supervised on a weekly basis. Neither the supervisors nor the therapists received feedback regarding their adherence ratings.
Statistical analysis
The inter-rater reliability coefficients were determined by calculating the intraclass correlation coefficient (ICC) using Model 2 [ICC(2,1)], with ICCs exceeding 0.75 considered good. Item 4 includes 15 session-specific items. Each session-specific item includes four to seven sub-items, depending on the content of the treatment session. We calculated the ICC for item 4 based on the mean value of all session-specific items. The ICC for the total adherence score was calculated based on the total sum score of items 1–9. Cronbach's α coefficient was used to test the internal consistency of all items using a 3-point Likert scale (items 5 and 6 were excluded due to the different scaling that resulted from their orientation on the original CPT Therapists’ Adherence Protocol; item 10 was also excluded). All of the analyses were conducted using IBM SPSS, version 22.
Results
Items 1–3 and 5–10 were rated in every session. The contents of item 4 varied depending on the respective session number. The frequency with which each CPT session was rated is presented in Table 1. The inter-rater reliability for the total adherence score was high, with ICC = 0.95. All ICCs and the mean, standard deviation (SD) and range for all items and for the expert ratings are presented in Table 2.
Table 1. Frequency with which each session had been rated within the 30 ratings
Table 2. Intraclass correlation coefficient, range, mean and standard deviation of items and mean and standard deviation of expert ratings for the CPT Adherence Rating Scale
ICC(2,1) = intraclass correlation coefficients for both raters; Min = lowest score on the ratings on a scale from 0 to 2 for items 1, 2, 3, 4, 7, 8 and 9; 1 to 5 for item 5; 0 to 1 for item 6, and 0 to 6 for item 10; Max = highest score on a scale from 0 to 2 for items 1, 2, 3, 4, 7, 8 and 9; 1 to 5 for item 5; 0 to 1 for item 6, and 0 to 6 for item 10. Relevance and appropriateness were assessed on a scale ranging from 0 to 3. *p < .001.
The Cronbach's α coefficient was .56. Three items generated low coefficient values and contributed to a slightly lower total score: item 7 (cognitive approach and reference to the PTSD disorder model; α = .60), item 8 (time management α = .65) and item 9 (implementation of interventions from treatments other than CPT; α = .61). However, no item was removed from the scale because of each item's importance with respect to its content and similarity to the original CPT Therapists’ Adherence Protocol. The three experts considered all the items relevant and appropriate, with M = 2.60 (SD = 0.15; range: 1–3) for relevance and M = 2.61 (SD = 0.37; range 1–3) for appropriateness.
Discussion
The results indicate that the newly developed ARS-CPT for our adapted CPT with more flexible sessions for patients with PTSD after CSA/CPA and borderline personality symptoms demonstrates good psychometric properties and is relevant and useful for assessing therapeutic adherence in future PTSD treatment studies with modified CPT versions using video ratings.
The results indicate excellent rater accordance for the entire scale and the individual items. Explanations for the particularly high ICCs of 1 could be the clear instructions in the CPT manual, the restricted response scales and the high familiarity of the raters with the ARS-CPT.
The internal consistency of the ARS-CPT was less satisfactory than expected, particularly for several items. One reason for the relatively low consistency score might be the restricted variance in the small and homogeneous sample. Because Cronbach's α depends on the length of the scale (Cortina, Reference Cortina1993), for the seven items using a 3-point Likert scale, the internal consistency was acceptable (Cortina, Reference Cortina1993).
In future trials, it might be helpful to assess item 4 (‘Implementation of intended interventions’) using distinct items to avoid being forced to compute a mean value that might have reduced internal consistency. Another advantage of using distinct items could be that session-specific elements, which are now mostly collapsed into one item (item 4), may contribute more to the total ARS-CPT score.
Additionally, using only one Likert scale for all items would extend the items considered for Cronbach's α and might augment its value but at the cost of deviating from the original CPT Therapists’ Adherence Protocol. Another important point is that other intervention studies have obtained similar results for Cronbach's α (e.g. Gutermann et al., Reference Gutermann, Schreiber, Matulis, Stangier, Rosner and Steil2015), which raises the question whether Cronbach's α is applicable to an adherence measure that assesses related but independent therapist actions rather than a single, unidimensional construct.
The ARS-CPT seems to have good content validity because the experts considered all of the items to be relevant and appropriate. In addition, the ratings were completed in an appropriate time, with an average completion time of 10 min.
Limitations
Repeated ratings by the same therapist treating the same patient may have artificially increased the inter-rater concordance in our study. Another limitation is the use of four different Likert response scales, which, for example, restricted the items considered for the internal consistency evaluation. The decision to remain as close to the original CPT Therapists’ Adherence Protocol as possible resulted in the use of different response scales (e.g. a 2-point Likert scale for item 6 and a 5-point Likert scale for item 5). For the newly developed items, we used a 3-point Likert response scale oriented on the response scales used in previous studies that assess global aspects of therapeutic adherence (e.g. Gutermann et al., Reference Gutermann, Schreiber, Matulis, Stangier, Rosner and Steil2015).
The sample of 30 videotapes was small. Therefore, the results should be confirmed using a larger sample of patients and therapists, which also would facilitate additional analysis, such as regarding the possible factor structure and the predictive value of the scale. In addition, the association of the presented scale with an assessment of therapeutic competence should be examined in future studies to ensure that each measure different aspects. In particular, item 5 (‘Established rapport’, ‘Reviewed the homework and discussed barriers to completing’ and ‘Structured the session and used time effectively’) might be confounded with therapeutic competence.
An additional limitation concerns the independence of the raters and experts. Both raters and two experts were involved in the treatment study because our adaption of the original CPT protocol is a relatively new treatment and only a few experts had full insight into its structure. Moreover, the raters were highly trained, which is time consuming and costly. The need to extensively train raters might impede the scale's future use. Nonetheless, for adherence assessment, an excellent knowledge of the manual is indispensable (Barber et al., Reference Barber, Triffleman and Marmar2007). Considering the findings in previous studies that non-experts could also assess adherence (Weck et al., Reference Weck, Weigel, Richtberg and Stangier2011b), future studies should examine whether raters who were not involved in the scale's development would also achieve such high rater concordance.
Furthermore, the transferability of the results is limited by the homogeneity of our patient sample because all of the patients were female and most had experienced CSA. This characteristic could also result in restricted variance, which could affect the statistical analysis.
Finally, another limitation could be related to the aim of our study, which was to create an adherence rating scale that could be used flexibly to assess different ways of administering CPT. One disadvantage of this approach could be that the scale does not include items that refer to specific problems resulting from frequent co-morbidities of PTSD, such as BPD in our study. Future studies could aim to develop adherence items that focus more on the specific treatment elements that address PTSD patients’ co-morbidities, such as alcohol abuse, BPD or psychosis.
Apart from these limitations, our findings suggest that the ARS-CPT is a reliable, appropriate instrument for assessing therapeutic adherence, which is important for future trials with CPT. The ARS-CPT instrument will be helpful for accurately interpreting the results of these studies, analysing possible effects of adherence on treatment outcomes and improving therapeutic skills training.
Acknowledgements
Ethics statement: The authors assert that all procedures included in this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975 and its most recent revision.
Conflicts of interest: Clara Dittmann, Meike Müller-Engelmann, Patricia A. Resick, Jana Gutermann, Ulrich Stangier, Kathlen Priebe, Thomas Fydrich, Petra Ludäscher, Julia Herzog and Regina Steil have no conflicts of interest with respect to this publication.
Financial support: The Release study was funded by the German Federal Ministry of Education and Research (01KR1303A). M.M.-E., P.A.R., K.P and R.S. received honoraria for providing supervision, workshops and presentations on PTSD treatments.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/S1352465816000679
Comments
No Comments have been published for this article.