Introduction
It is crucial that there are high-quality measures of client change processes in order to successfully study psychotherapy change mechanisms (Kazdin, Reference Kazdin2007; Lutz and Hill, Reference Lutz and Hill2009). There are a large number of diverse measures that have been utilized in the varied fields of psychotherapy process research (Llewelyn and Hardy, Reference Llewelyn and Hardy2001). We report the development of a new observer measure of client change processes informed by perceptual control theory (PCT; Powers, Reference Powers1973, Reference Powers2005; Powers et al., Reference Powers, Clark and McFarland1960a,b). We shall explain this theoretical approach to client change, illustrate its relevance to existing conceptualizations, and describe the development and validation of the measure. We propose that the early development of this measure is important to report for a number of reasons: (1) it appears to be the only observer measure available that is exclusively focused on the transtheoretical concept of the client's current awareness (Higginson and Mansell, Reference Higginson and Mansell2016a); (2) PCT is being increasingly used to inform the development of psychological interventions including Method of Levels therapy (MOL; Carey, Reference Carey2006; Alsawy et al., Reference Alsawy, Mansell, Carey, McEvoy and Tai2014), group courses (Morris et al., Reference Morris, Mansell and McEvoy2016), and digital applications (Gaffney et al., Reference Gaffney, Mansell, Edwards and Wright2014; Carey et al., Reference Carey, Haviland, Tai, Vanags and Mansell2016), thereby making it important to build the methodological foundations of how to measure their potential mechanism of change; (3) the measure lends itself to a dynamic graphic representation of client awareness during a session that can be utilized in evaluating MOL and its efficiency at targeting a proposed mechanism of psychological change (see https://youtu.be/SaDmLRJ2DGY); and (4) it may promote further research to refine and adapt the measure.
PCT is an integrative mechanistic framework that proposes a transdiagnostic explanation for psychological functioning, distress and recovery. PCT provides a mechanistic and functional account of both ‘normal’ and ‘abnormal’ human functioning and behaviour. It is not based on the language or terminology of any one particular school of psychological theory. PCT has informed a form of cognitive therapy known as Method of Levels (Carey, Reference Carey2006), which has a variety of advantages: (1) it can be applied across diagnostic categories (Carey, Reference Carey2008); (2) it is associated in open trials with large effect sizes for reductions in anxiety, depression and stress (Carey et al., Reference Carey, Carey, Mullan, Spratt and Spratt2009); (3) it may be more efficient than other psychological therapies in terms of improvements per session (Carey et al., Reference Carey, Tai and Stiles2013); and (4) it can be provided on an ad hoc basis as a drop-in service within contexts such as in patient wards, schools and prisons (Tai, Reference Tai2017).
The main processes involved in psychological functioning and distress according to PCT are described below.
Control
PCT regards individuals as sophisticated living control systems who strive to control their perceptions to ensure their experience is a certain way. Control systems operate by detecting differences between the goal specified by the control system and what is perceived. When differences are perceived, an individual acts in a way that will reduce this so that perceptions remain consistent with goals (Powers, Reference Powers1973, Reference Powers2005).
Hierarchies
The way in which the numerous goals held by a person are continually met is through the hierarchical arrangement of control systems. Higher level, long-term goals (for example, the self-concept of being a good person) leads to the setting of subgoals for an individual's principles (e.g. to be helpful), which in turn regulate lower level, shorter term goals (e.g. to move closer to a friend).
Conflict
PCT suggests that psychological distress arises from the loss of control caused by conflict between control system goals. This occurs when an individual is striving to reach two incompatible goals at the same time (Powers et al., Reference Powers, Clark and McFarland1960a,b). According to PCT, conflict involves at least three levels of the control system hierarchy. At the highest level involved in a conflict, information sent to the next level down may result in incompatible goals being set. The conflicted goals at this level therefore in turn send signals to the level below that inevitably vacillate and vary unpredictably. For example, a person may have a higher level goal to ‘be a good parent’. This goal then sets conflicting goals at the level below which include ‘keep daughter safe’ and ‘let daughter be normal’. If the parent ‘keeps the daughter safe’, this generates discrepancy around the goal of ‘letting the daughter be normal’. However, if the parent meets the goal of ‘letting the daughter be normal’, this generates discrepancy around the goal of ‘keeping the daughter safe’. These incompatible goals in turn lead to a number of further conflicting goals at the level below aimed at meeting both goals such as ‘let her go out’, ‘keep her home’, ‘be vigilant’, ‘be relaxed’, etc. (see Fig. 1).
Figure 1. A PCT model of conflict between control system goals over three levels. Reproduced with permission from Carey (Reference Carey2006).
Conflict may be maintained by a process termed arbitrary control. Arbitrary control occurs when a person controls an experience in order to meet one goal but does not take into account another goal that is also controlling the same experience. This could come about because the person is either unaware or, at that time, unconvinced of its relative importance (Mansell, Reference Mansell2005; Powers, Reference Powers1973). In the example above, the parent may prevent the daughter from going out so as to keep the daughter safe, without regard for the other goal of letting the daughter be normal. Thus arbitrary control strategies are typically counterproductive and inflexible strategies that compromise functioning, as described earlier.
Awareness and change
According to PCT, a fundamental process necessary for the resolution of conflict and thus psychological distress is that of sustained awareness on the higher level control systems which are responsible for the conflict between goals. The reason sustained awareness is necessary is because it allows a process called reorganization to occur at the appropriate location for lasting change. Reorganization is an automatic learning process that reduces the loss of control caused by conflict between goals. It randomly changes the properties of the control systems responsible for conflict until such conflict is reduced (Mansell, Reference Mansell2005; Powers, Reference Powers1973). The process of reorganization has been successfully modelled (Marken and Powers, Reference Marken and Powers1989; Powers, Reference Powers2008), and the concept of reorganization has been used to explain the effectiveness of the common therapeutic technique of exposure (Carey, Reference Carey2011). According to PCT, encouraging present moment awareness helps individuals to identify and reduce arbitrary control processes that limit awareness of goal conflicts so that a ‘window’ of awareness can be brought to bear on conflicted goals (Carey, Reference Carey2006; Powers et al., Reference Powers, Clark and McFarland1960b). This is thought to be the mechanism of change within effective psychological therapies, that is directly addressed within MOL.
What is important to note is that according to PCT, the process of reorganization will only successfully reduce conflict and thus psychological distress over the long term when:
(1) awareness is controlled and thereby sustained for a long enough period of time, and
(2) at the level of the control system hierarchy responsible for conflict, i.e. at the control system that is setting the goals for the two conflicting control systems at the level below (Powers, Reference Powers1973; Carey, Reference Carey2006).
The primary aim of this study was to investigate the psychometric properties of a new measure of client awareness based on PCT. We examined its inter-rater reliability and to some degree, its convergent validity. Specifically, we hypothesized that the duration of sustained awareness and the frequency of shifts in depth of awareness (as measured by the scale) would be:
(1) positively correlated with a global and subjective measure of awareness and therapeutic change;
(2) positively correlated with self-report measures of processes associated with psychological change. We did not identify any criterion measures of the same constructs as those assessed by the scale so we used distally related measures and anticipated a modest correlation;
(3) positively correlated with changes in symptoms between the session rated and the following session.
Method
Participants
Data collected from a Method of Levels open trial in NHS Primary Care Mental Health services was utilized for the current study (Lansbergen, Reference Lansbergen2011). Sixty-three participants over the age of 16 years experiencing mental health problems of mild to moderate severity who had been referred to an NHS Primary Care Mental Health Service participated in the trial. Out of the total of 63 participants who took part in the trial, 50 first sessions were audio recorded and transcribed verbatim, and were thus available for coding. Ethical approval for the MOL open trial was obtained from a local NHS Research Ethics Committee. NHS Trust Research and Development approval was obtained from participating local NHS Trusts.
Measures
We selected three pre-existing self-report measures of psychological change, described below, to attempt to cross-validate our new measure of client awareness.
Reorganisation of Conflict Scale (RoC; Higginson, Reference Higginson2007)
The RoC is a 22-item self-report questionnaire that assesses a person's tendency to engage in processes that either promote or block the process of reorganization. It therefore has the same theoretical basis as the D-DACS, making it appropriate for cross-validation with the caveat that it was measured from a different perspective (the client's own self-report). Factor analysis of the scale suggests that there are three underlying subscales: ‘components of goal conflict reorganization’, ‘inflexible and urgent problem solving’, and ‘awareness of goal conflict’. Preliminary exploration of the internal reliability of the subscales indicates that the ‘components of goal conflict reorganisation’ subscale has good internal reliability (α = .83), whereas ‘inflexible and urgent problem solving’ and ‘awareness of goal conflict’ both fall short of acceptable values. ‘Inflexible and urgent problem solving’ and ‘awareness of goal conflict’ both demonstrated significant weak to moderate relationships with a transdiagnostic measure of maintenance processes, and have been found to be significant predictors of moderate symptom levels of depression. ‘Components of goal conflict reorganization’ has been found to be a significant predictor of not meeting criteria for moderate anxiety (Bird, Reference Bird2013). Within both non-clinical (n = 294) and clinical (n = 156) samples, the subscales have shown good convergent validity with measures of psychological inflexibility (including experiential avoidance), intolerance of uncertainty and mindfulness (Morris et al., Reference Morris, Tai, Bird, Higginson and Mansell2017).
University of Rhode Island Change Assessment Scale (URICA; McConnaughy et al., Reference McConnaughy, Prochaska and Velicer1983)
The URICA is a 32-item self-report scale measuring motivation to change. Although it has a different theoretical basis from the D-DACS, it is widely used and validated, and many of its questions relate directly to decisions regarding whether to attend to or talk about a problem, as well as the capacity to contemplate ambivalence (conflict) regarding this. Statements are rated (e.g. ‘it might be worthwhile to work on my problem’) on a scale of 1 (strongly disagree) to 5 (strongly agree). The scale has four subscales (Precontemplation, Contemplation, Action and Maintenance) representing different stages of change, along with a total readiness to change score. Internal consistency has been found to be good, with alphas ranging from 0.79 to 0.89 for the four subscales (McConnaughy et al., Reference McConnaughy, Prochaska and Velicer1983, Reference McConnaughy, Diclemente, Prochaska and Velicer1989). The scale has been shown to have a significant, but modest, predictive validity, for example when predicting drop-out rates in exposure therapy (Solem et al., Reference Solem, Husby, Håland, Launes, Hansen, Vogel and Hagen2016).
Readiness for Cognitive Change Scale (RCCS; Lansbergen, Reference Lansbergen2011)
The RCCS is an 8-item scale assessing readiness to talk about, take responsibility for, and change the way they view their problems using a 5-point Likert scale. Alpha in a UK clinical sample was .67 (with item 4 removed) (Kelly, Reference Kelly2011). In terms of validation, a multi-level analysis of MOL showed that the RCCS at the start of a session predicted subsequent reductions in anxiety and depression as assessed at the start of the subsequent session (Kelly et al., Reference Kelly, Lansbergen, Samad, Carey, Tai and Mansell2011). We anticipated that clients who were willing and able to talk about their problems and shift their view of them, would be those for whom the D-DACS indicated they had shifted and sustained their awareness to aspects of the problem measured by this scale (e.g. present moment experiences, arbitrary control, conflict).
The Generalized Anxiety Disorder-7 (GAD-7; Spitzer et al., Reference Spitzer, Kroenke, Williams and Lowe2006)
The GAD-7 is a measure of psychological wellbeing used routinely in Primary Care Mental Health Services to collect service data for the Improving Access to Psychological Therapies (IAPT) programme (Department of Health, 2008). It is a 7-item self-report questionnaire measuring symptoms of generalized anxiety over the previous 2 weeks. Each item is rated according to its frequency of occurrence on a 4-point scale of 0 (not at all) to 3 (nearly every day). The GAD-7 has a maximum total score of 21, and cut-off points of 5, 10 and 15 represent mild, moderate and severe levels of anxiety, respectively. The GAD-7 has good internal consistency with a Cronbach's alpha of .92 (Spitzer et al., Reference Spitzer, Kroenke, Williams and Lowe2006).
The Patient Health Questionnaire-9 (PHQ-9; Kroenke et al., Reference Kroenke, Spitzer and Williams2001)
The PHQ-9 is a measure of psychological wellbeing used routinely in Primary Care Mental Health Services to collect service data for the IAPT programme (Department of Health, 2008). It is a 9-item self-report questionnaire of depression severity. Each item is rated according to its frequency of occurrence over the past 2 weeks using a 4-point scale of 0 (not at all) to 3 (nearly every day). The PHQ-9 has a maximum total score of 27, and cut-off points of 5, 10, 15 and 20 represent mild, moderate, moderately severe and severe depression, respectively. The PHQ-9 has been found to have excellent internal reliability with Cronbach's alpha values ranging from .86 to .89 (Kroenke et al., Reference Kroenke, Spitzer and Williams2001).
The D-DACS
The D-DACS (Higginson and Mansell, Reference Higginson and Mansell2016b) is an observer-rated within-session coding scheme developed for the measurement of the core within-session client change processes of shifts in depth of awareness and duration of awareness. Although client self-report measures are relatively inexpensive and can be easily incorporated in study procedures (Amato et al., Reference Amato, Hofferth and Casper2007), they are somewhat limited in their usefulness for therapy process research due to a low level of complexity in the constructs measured and low accuracy (Baumeister et al., Reference Baumeister, Vohs and Funder2007). Observer-rated measures offer an increased complexity in their measureable constructs (Kramer, Reference Kramer2008) and also allow the measurement of processes that might be limited in their availability for self-reflection by a client. Hill and Lambert (Reference Hill, Lambert and Lambert2004) state that the chosen unit of analysis needs to match the construct under investigation. We therefore selected a microprocess measure to focus on small units such as words, phrases and sentences. However, to establish internal validity we included a macroprocess measure of a whole session, which required the coders to make a global judgement. We also needed to establish the type of measurement. Interval scales in which constructs are rated on Likert scale are advantageous for data analytic reasons as data can be averaged across coders. Nominal category systems, in which judgements are made about the absence or presence of a target event, often serve to reflect phenomena more accurately (Hill and Lambert, Reference Hill, Lambert and Lambert2004). We established that a measure that incorporates both forms of measurement would benefit from the advantages of both.
We sought feedback from multiple perspectives from the inception of the scale: individuals with both research and clinical expertise in PCT and MOL, an author of an existing therapy process coding scheme and a novice coder, to ensure it was possible to successfully train others in the use of the coding scheme. During these piloting stages, the coding scheme was modified, elaborated and adapted. First, to ensure fidelity to its theoretical underpinnings, it was necessary that the D-DACS would have the facility to code both the duration of awareness and also the depth of awareness. This led the initial conceptualizations of the coding scheme to include two orthogonal dimensions. In line with PCT's idea that reorganization is an inevitable consequence of sustained awareness (where there is conflict), the duration of awareness dimension had two subcategories: sustained awareness and reorganization. Furthermore, in accordance with PCT's conceptualization of conflict as involving at least three levels of the control system hierarchy, the depth of awareness dimension was further specified as encompassing three subcategories: higher level goals, goal conflict and arbitrary control. Following further feedback, a fourth important component of client awareness – present moment perceptual experience – was added to the coding scheme. During piloting, it was observed that clients would frequently shift their awareness very briefly, almost transiently, to certain content. To enable enquiry into whether such fleeting shifts in awareness were a significant therapeutic change process, it was decided that instances of fleeting shifts in awareness were important to capture within the coding scheme. Therefore a further category within the duration of awareness dimension – fleeting awareness – was added. Further feedback highlighted that PCT proposes that awareness needs to be sustained for long enough that reorganization can occur, and so the duration of time spent at higher levels would be theoretically important to capture. Therefore, it was decided that duration of sustained awareness as well as the frequency of shifts in depth of awareness would be the primary indices of awareness produced by the coding scheme.
The final twelve codes of the D-DACS and corresponding descriptions are presented in Table 1. The coding manual (Higginson and Mansell, Reference Higginson and Mansell2016b) includes the operational definitions of each category as well as real examples to facilitate training and agreement among raters. Figure 2 provides a visual illustration of how the four components of awareness coded by the D-DACS map onto the control hierarchy of PCT. Figure 3 uses a hypothetical clinical example to demonstrate what a client might say in a therapy session when awareness is directed at each of the four D-DACS components.
Table 1. The finalised D-DACS codes
Figure 2. Diagram representing how awareness can be directed to four components within a conflicted control system
Figure 3. A diagrammatic representation of awareness at four levels with the control system hierarchy using a hypothetical case illustration
The review and early development of the D-DACS led to the requirement of an orthogonal system encompassing two distinct dimensions of awareness:
(1) duration of awareness – capturing the shifting to and sustaining of awareness to systems responsible for conflict, allowing reorganisation to occur;
(2) shifts in depth of awareness capturing the four depths of awareness involved in conflict:
• present moment perceptual experience of thoughts, feelings, images, etc.;
• awareness of counterproductive arbitrary control strategies;
• awareness of goal conflict;
• awareness of the higher level goals driving conflict.
Shifts in depth of awareness to present moment perceptual experience, arbitrary control processes, goal conflict and higher order goals are coded in the D-DACS. The duration of awareness including transient shifts, sustained awareness and awareness incorporating components of reorganization, are also coded by the D-DACS. For details of the D-DACS manual, see Higginson and Mansell (Reference Higginson and Mansell2016b).
Primary indices of awareness include:
• Shifts in depth of awareness – the frequency with which shifts in awareness occur to the four levels of the D-DACS hierarchy.
• Duration of sustained awareness – the amount of time awareness is sustained at the four levels of the D-DACS hierarchy.
Global indices of awareness include:
• Global index of depth of awareness – a coder's overall impression of level of the D-DACS hierarchy awareness reached in a session (independent of duration of awareness).
• Global index of duration of awareness – a coder's overall impression of the degree to which awareness was sustained and reorganization occurred at any of the four levels of the hierarchy (independent of the level of hierarchy awareness reached)
• Global index of overall therapeutic progress – a coder's overall impression of both the degree to which a participant sustained awareness and reorganization conflict during a session and the level of hierarchy the participant's awareness reached in a session.
The coding procedure was as follows. First, the coder listened to the entire therapy session without pausing then rated the global indices of awareness. The coder then listened to the session again and coded instances of the primary indices of awareness. During this stage the audio tape could be replayed as many times as necessary to determine the appropriate code. For instances of sustained awareness, the coder indicated on the transcript when the period of sustained awareness began and ended, so that the duration of sustained awareness could be calculated. The coder could listen to the session as many times as necessary until satisfied that all instances of categories had been coded. Once the coder had coded the session according to the primary indices of awareness, the coder then re-coded the global indices of awareness again.
Method of Levels
A form of cognitive therapy directly informed by the principles of PCT, known as Method of Levels (Carey, Reference Carey2006) was used in the current study. MOL involves helping the client to talk about their problem in the present moment and to help them notice disruptions in their flow of dialogue indicative of shifts in awareness at higher levels of the goal hierarchy. The role of the therapist is to help the client sustain their awareness on personally valued goals. Sustained awareness enables an individual to restore control over their life through becoming aware of and letting go of habitual patterns of inflexible and arbitrary control over their thoughts, feelings and behaviour (Mansell, Reference Mansell and Dryden2012). In turn it is this process that enables individuals to evaluate, reprioritize and pursue their over-arching personal goals to improve their quality of life. Thus MOL harnesses the key transdiagnostic processes thought to be involved in therapeutic change highlighted earlier. The first MOL session with participants lasted for 1 hour. Participants were first required to complete the self-report measures and the remaining time was then used to conduct MOL. For that reason, participants received varying lengths of MOL.
Therapists
Five therapists delivered MOL in the current study; one therapist was a qualified clinical psychologist with training and expertise in MOL. The remaining therapists were a trainee clinical psychologist and three research assistants who had all attended an externally validated MOL training workshop prior to the study. Weekly supervision continued throughout the course of the trial.
Analysis plan
Independent t-tests (for continuous variables) and chi-squared tests (for categorical variables) were used to assess differences between participants who attended one MOL session and participants who returned for a second. All variables were screened for normality using the Kolmogorov–Smirnov test. Despite efforts, variables remained non-normally distributed after transformations were applied. Therefore parametric analyses were undertaken where analysis included normally distributed variables and non-parametric equivalents were used where analyses included variables that had not been transformed successfully. The convergent validity of the D-DACS primary indices of awareness were assessed by correlations with the D-DACS global indices of awareness and a self-report measures of conceptually related processes. The predictive validity of the D-DACS primary indices was assessed by correlations with changes in self-report symptoms of depression and anxiety (PHQ-9 and GAD-7) between sessions 1 and 2 (for participants who attended a second session). Exploratory analysis included considering associations between the primary indices of awareness and measures of other theoretically related constructs (URICA, RCCS) and the primary subindices of awareness and self-report measures.
Results
Sample characteristics
Demographic information and self-report measures for the sample overall, for those who returned for a second session and for those who did not, are presented in Table 2. Comparisons indicated that participants who returned for a second session did not significantly differ in terms of gender, ethnicity, length of first MOL session, pre-therapy depression symptoms, pre-therapy anxiety symptoms, RoC scores or RCCS scores. Participants who returned for a second session were found to be significantly older than participants who did not. Scores on the precontemplation subscale of the URICA were significantly higher in participants who returned for a second session compared with those who did not, suggesting that participants who returned for a second session were significantly more likely to be unaware of a problem and unwilling to change. Participants did not differ on the remaining subscales of the URICA. For participants who returned for a second session, there was a significant difference found between participants scores on the PHQ-9 between sessions 1 (mean = 17.25, SD = 6.37) and 2 (mean = 14.53, SD = 7.02) (t [31] = 3.14, p < 0.01). There was also a significant difference found between participants’ scores on the GAD-7 between sessions 1 (mean = 14.63, SD = 5.2) and 2 (mean = 12.32, SD = 6.03) (t [31] = 2.63, p < 0.05).
Table 2. Sample characteristics
*p < 0.05; **p < 0.01; n.s., not significant.
The number of shifts in awareness divided by the categories of the D-DACS are shown in Table 3.
Table 3. Median, inter-quartile ranges, minimum and maximum values and tests of normality for the shifts in depth of awareness indices
*Significant at p < 0.05; ***significant at p < 0.001. IQR, interquartile range; K–S, Kolmogorov–Smirnov.
As Table 4 shows, awareness was sustained at any of the four components of the D-DACS coding system for an average of 384.70 s per session, corresponding to an average of 22.3% of total session length. Awareness was sustained at the level of arbitrary control processes more so than at any other level of the D-DACS hierarchy. Awareness was sustained the least at the level of higher order goals. Awareness at the level of present moment perceptual experience and goal conflict was sustained for a comparable amount of time.
Table 4. Means, standard deviations, minimum and maximum values in seconds and tests of normality for the duration of sustained awareness indices
The coding of the primary indices of awareness did not have a significant impact on the subsequent re-coding of the global indices of awareness, and so the global indices of awareness coded prior to the coding of the primary indices were used in analysis.Footnote 1
Inter-rater reliability
Twenty five per cent of the sample (12 participants) were randomly selected and coded by the second author for the purposes of inter-rater reliability analysis. Transcripts were given to the second author with the passages coded by the first author already demarcated but with the codes omitted. The second author then independently listened to the MOL session and provided the codes judged to correspond to each demarcated passage.
The percentage of agreement of depth of awareness codes (80.7%), duration of awareness codes (79.9%), and all 12 codes (66.3%) was substantial. There was a substantial amount of agreement (Landis and Koch, Reference Landis and Koch1977) between coders when judging the depth of the D-DACS hierarchy a participant's awareness was directed (κ = .71) and when judging the duration of awareness (κ = .64). However, we did not calculate κ for all 12 codes owing to the relative number of observations to the number of possible codes which an independent university statistician advised would be unreliable to calculate and report.
Evaluating the validity of the D-DACS primary indices of awareness
Convergent validity
Moderate positive correlations between the primary and global indices of awareness (made prior to coding the primary indices) were found, indicating that the D-DACS primary indices of awareness demonstrate a good level of convergent validity with the more subjective, index of awareness as measured by the D-DACS (see Table 5). There were no significant correlations between the D-DACS primary indices of awareness and any of the self-report scales or subscales (all r s < .3, not significant). This suggests that the D-DACS lacked convergent validity with a standardized self-report measure of conceptually related processes.
Table 5. Correlation coefficients between primary indices and global indices
**Significant at p < 0.01.
Predictive validity
No significant positive correlations were found between the primary indices of awareness and changes in PHQ-9 and GAD-7 scores between sessions 1 and 2 for those participants who returned for a second session. This suggests that the D-DACS did not predict changes in symptoms of depression and anxiety between the session coded by the D-DACS and the subsequent session.
Discussion
The current study explored the psychometric properties of a novel observer-rated within-session coding scheme for measuring core transdiagnostic and transtheoretical therapy change processes: the Depth and Duration of Awareness Coding Scheme. The scale indicated good convergent validity with subjective assessments of awareness and therapeutic change coded from the same session. It did not converge with conceptually related, but distinct, self-report scales thought to measure the potential for therapeutic change, or with actual symptom change as measured by the shift in symptoms leading up to the following therapy session. The D-DACS showed a good level of inter-rater reliability.
The findings suggest that the D-DACS primary indices did not demonstrate predictive validity in relation to the prediction of changes in symptoms. This finding may suggest that the D-DACS primary indices of awareness are not valid measures of client change processes. However, it is also possible that these findings are due to the choice of outcome measure and duration of outcome assessed. The assessment of outcome based on changes in levels of distress may have been more consistent with the predictions of PCT as opposed to the assessment of outcome based on changes in levels of symptoms. It is also possible that the duration of outcome assessed (just one between session interval) was not long enough to expect that changes in symptoms would occur.
Exploration of the inter-rater reliability of the depth and duration of awareness codes considered separately revealed a good level of agreement between coders. The amount of agreement on the depth and duration of awareness codes when considered together, reduced to 66.25% of agreement. However, this figure does not take into account the amount of chance agreement. Moreover, the second coder was restricted to code the passages that had already been selected by the first coder. The establishment of inter-rater reliability when both coders are free to code sections of the transcript judged to be relevant would represent a more rigorous assessment of the inter-rater reliability of the D-DACS.
Consideration of the descriptive data on the D-DACS suggest that whilst participants shifted the depth of awareness most frequently to present moment perceptual experience, awareness was not necessarily subsequently sustained there. Shifts in awareness to arbitrary control processes occurred less frequently but participants sustained their awareness at this level for longer than that of present moment perceptual experience or any other component of the D-DACS. This finding may be due to training and experience of two of the MOL therapists in the use of cognitive behavioural interventions. It is possible that these therapists were accustomed to talking to clients about cognitive and behavioural processes that would likely be coded as awareness of arbitrary control processes.
Shifts in awareness towards higher order goals were least frequent and did not last as long as the other categories. This is consistent with what might be expected of a first session MOL therapy. It is likely that as therapy progresses a client progressively shifts towards higher order goals and explores them for longer. Nevertheless, the presence of floor effects represents a limitation to the current study in terms of the ability of the D-DACS to demonstrate predictive validity.
There are a number of limitations to the current study which should be considered when interpreting its findings. The study explored the psychometric properties of a newly developed coding scheme, very much in its infancy in terms of expertise and experience in its use. It is likely that with further experience and training, coders will become more proficient and reliable in the coding of the processes measured by the D-DACS.
The current study utilized participants’ first MOL sessions. Due both to the study procedure and standard MOL delivery (in which session lengths are controlled by the client), the length of sessions in the current study varied considerably (the shortest session being just 19 minutes). Therefore, it is possible that there was a low frequency in occurrence of shifts in and periods of sustained awareness representing a floor effect, in turn limiting the validity of the current study.
Therapy processes were sampled in one first therapy session and used to predict outcome in one between-session interval. This may have limited the extent to which the validity of the D-DACS could be demonstrated. Establishment of the validity of other existing observer-rated within-session therapy process measures have utilized a diverse range of study designs, sampling a wide range of therapy samples including whole therapy sessions early, middle or late in therapy (Kramer et al., Reference Kramer, Despland, Michel, Drapeau and De Roten2010), therapy segments of varying lengths (Goldman et al., Reference Goldman, Greenberg and Pos2005), all therapy sessions for a small sample of participants (Goncalves et al., Reference Goncalves, Mendes, Cruz, Ribeiro, Sousa, Angus and Greenberg2012) and therapy sessions which preceded a significant drop in symptoms (Tang et al., Reference Tang, DeRubeis, Beberman and Pham2005).
The study has highlighted a limitation to the D-DACS in its current form. The D-DACS codes the frequency of shifts in depth of and duration of awareness, regardless of the content of awareness. According to PCT the longer awareness is sustained on the particular control system driving conflict the more likely that effective reorganization will occur. Therefore, PCT would predict that shifting and sustaining awareness for a long period on one or a small number of key conflicts would be more helpful than shifting and sustaining awareness on numerous conflicts within a session. Currently, the D-DACS does not have a way of capturing whether a participant's awareness has been focused on a particular conflicted system or numerous conflicted systems. This represents a current limitation of the D-DACS as it stands.
Future research aiming to establish the psychometric properties of the D-DACS should endeavour to expand and formalize the training procedure of D-DACS coders to improve inter-rater reliability, modify the inter-rater reliability analysis procedure in order to assess the reliability of the D-DACS when both coders are free to select relevant passages, consider the inclusion of other observer-rated within-session measures of conceptually related change processes by which the validity of the D-DACS could be assessed against, and consider the use of alternative study designs such as those in the studies cited above in order to increase the sampling of change processes. Further development of the D-DACS should consider its capacity to code the extent to which awareness is focused on key conflicts or numerous conflicts within a session.
Conclusions
The current study found preliminary support for the inter-rater reliability of the D-DACS. Support for the convergent and predictive validity of the D-DAS was not found. If the limitations of both the coding scheme itself and the current study design were addressed in future studies, the validity of the D-DACS as a measure of core client change processes could be more firmly established.
Main points
(1) A new measure, the Depth and Duration of Awareness Scale (D-DACS), was constructed.
(2) The D-DACS showed good reliability and mixed evidence of validity.
(3) The study provides the preliminary work for future development and refinement of the measure.
Conflicts of interest
The authors declare no conflicts of interest with respect to this publication.
Ethical statement
The authors have abided by the Ethical Principles of Psychologists and Code of Conduct as set out by the APA. The Ethics Reference Number is REC 09/H1016/105.
Funding
The open trial was partly funded by the ESRC (RES-060-25-0044).
Learning objectives
(1) To consider the evidence for whether the change process across psychological therapies is the depth and duration of awareness (attention).
(2) To learn how perceptual control theory can inform a novel observer-coded measure of depth and duration of awareness.
(3) To consider the preliminary evidence for the reliability and validity of the new measure.
Comments
No Comments have been published for this article.