Social anxiety disorder (SAD) is the most common anxiety disorder and the third most prevalent psychiatric condition (Brunello et al., Reference Brunello, Den, Judd, Kasper, Kelsey, Lader and Wittchen2000), with prevalence rates between 5% and 13.3% (Kessler, Reference Kessler2003). SAD is characterized by persistent or strong fear in situations where the individual may feel embarrassed or afraid of acting or behaving in a humiliating or embarrassing way eventually causing anxiety and phobic avoidance (APA, 1994). SAD has drawn attention due to its high prevalence and it is considered to be a serious public health problem because of its underdiagnosis rate -only 3% of cases are properly detected (Davidson, Hughes, George, & Blazer, Reference Davidson, Hughes, George and Blazer1993)- and of the impairments and limitations it causes in several areas of daily life (Stein & Kean, Reference Stein and Kean2000; Wittchen, Fuestsch, Sonntag, Müller, & Liebowitz, Reference Wittchen, Fuetsch, Sonntag, Müller and Liebowitz2000).
Studies regarding the impact of SAD in daily life have demonstrated that it affects almost every aspect of one’s life, hampering the performance of daily activities and reducing financial resources, psychological well-being, self-esteem, interpersonal relationships and social participation (Crippa, Reference Crippa2009; Stein & Stein, Reference Stein and Stein2008).
Some authors underscored the necessity and importance of evaluating the impact of the disorder on daily functioning (Hambrick, Turk, Heimberg, Schneier, & Liebowitz, Reference Hambrick, Turk, Heimberg, Schneier and Liebowitz2003; Kessler, Reference Kessler2003; Safren, Heinberg, Brown, & Holle, Reference Safren, Heinberg, Brown and Holle1996; 1997; Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994; Stein & Kean, Reference Stein and Kean2000; Wittchen et al., Reference Wittchen, Fuetsch, Sonntag, Müller and Liebowitz2000). In these studies, SAD is presented as a disorder that strongly interferes with the ability to perform social roles as being a student, employee, friend, and family member, with further effects on self-care, self-maintenance activities, and leisure. Poor social functioning is the main criterion in the Diagnostic and Statistical Manual –4th edition (DSM-IV – APA, 1994)- and International Statistical Classification of Diseases and Related Health Problems– 10 (ICD-10 – WHO, 1993)- for the diagnosis of SAD.
Thus, with the aim of contributing to the identification of functional impairments related to SAD, a group of researchers proposed two scales to specifically assess the impact of SAD on normal daily activities, a self-rated instrument, originally called Liebowitz Disability Self-Rating Scale (LSRDS), and another clinician-rated, originally named Disability Profile/ Clinician-Rated (DP - Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994).
To validate the scales, the researchers evaluated 32 patients with SAD and 14 participants without psychiatric disorders, observing that SAD subjects presented moderate to severe impairment in areas including school, work, family relations, dating/marriage, friendship/social networks, and other interests. The results of this survey agree with data from another investigation that also pointed out to the association of SAD with severe obstacles for the performance of daily activities (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994). Additionally, the scales proved able to detect impairments associated with SAD in different ways, with good criterion validity and internal consistency. Nevertheless, these studies involved a small sample and important psychometric properties, such as concurrent validity, reliability, and factor analysis were not assessed. Moreover, as far as we know, the scale was not studied in other cultures and contexts, which restricts the generalization of the findings and justifies the proposition of a new psychometric study.
Therefore, our objective was to assess the reliability and validity of the Disability Profile/Clinician-Rated (DP) scale applied to a group of Brazilian university students with and without SAD.
Method
Participants
This study is part of a broader investigation carried out between 2004 and 2009, which aimed at systematically assessing epidemiological and clinical aspects of SAD. Self- and clinician-rated assessment instruments were used to evaluate a population of university students enrolled at several courses in two universities (one public and one private), in medium-sized cities from the northeastern region of the state of São Paulo, Brazil (Osório, Crippa, & Loureiro, Reference Osório, Crippa and Loureiro2006).
The participants included in the sample were systematically screened among 2613 university students aged 17–35 years. Participants treated using neuroleptic drugs and providing incomplete forms were excluded, as well as subjects with present or prior psychiatric co-morbidities including psychotic disorders, dependence on psychoactive substances, eating disorders, panic disorder, and recurrent depressive disorder. Given the high prevalence of co-morbidities associated with SAD (Filho et al., Reference Filho, Hetem, Ferrari, Trzesniak, Martín-Santos, Borduqui and Crippa2010), participants with a previous history of depressive episodes, generalized anxiety disorder, and simple phobia were included in the sample.
Assessments were performed by psychiatrists and psychologists with vast clinical experience and properly trained in the use of the above mentioned instruments and conduction of the interviews.
The study was approved by the Ethics Committee of the Ribeirão Preto Medical School University Hospital (n° HC-FMRP 11570/2003) and all participants signed informed consent forms to participate.
Instruments
The Social Phobia Inventory (SPIN) and the MINI Social Phobia Inventory (MS) are self-rated instruments designed to evaluate fear, avoidance, and physiological symptoms associated with SAD. The SPIN consists of 17 items scored on a 4-point Likert scale (Connor et al, Reference Connor, Davidson, Churchill, Sherwood, Foa and Weisler2000; Osório, Crippa, & Loureiro, Reference Osório, Crippa and Loureiro2009, 2010b) and the MS is a condensed form composed of items 6, 9, and 15 of the SPIN. The presence of SAD in the MS is indicated by scores equal to or higher than 6. Based on an empirical study (Connor, Koback, Churchill, Katzelnick, & Davidson, Reference Connor, Kobak, Churchill, Katzelnick and Davidson2001), this reduced form of the instrument showed a high discriminative power to detect SAD, with sensitivity of 88.7% and specificity of 90%. In the Brazilian setting, two studies assessed the psychometric properties of the MS, reporting sensitivity of 94%, specificity of 46%, and internal consistency, Cronbach’s alpha of .49-.73. For this study, a version of the MS translated and validated to Brazilian Portuguese was used (Osório, Crippa, & Loureiro, Reference Osório, Crippa and Loureiro2007, 2010a).
The Structured Clinical Interview for DSM-IV (SCID-IV, First, Spitzer, Gibbon, & Williams, Reference First, Spitzer, Gibbon and Williams1997), translated and adapted to Portuguese (Del Ben et al., Reference Del-Ben, Vilela, Crippa, Hallak, Labate and Zuardi2001), is frequently used to make psychiatric clinical diagnoses based on the DSM-IV by means of 10 modules that can be applied in an independent or combined manner, according to the objectives. For this study we used Module A –Mood Episodes; Module B -Psychotic and associated symptoms; Module C –Differential diagnoses of psychotic disorders; Module E –Disorders of use of alcohol and of other substances; Module F –anxiety disorders and other disorders
The Disability Profile/Clinician-Rated (DP) scale, rated by the investigator to measure functional impairment, is applied as a guide for the clinical assessment interview conducted by health professionals. The rater assesses the presence of functional impairment experienced by participants over the preceding two weeks and during lifetime attributed to the presence of SAD. The scale contemplates eight fields: school, work, family, marriage/dating, friendship, other interests (like going to clubs, attending sports and religious activities, and hobbies/leisure), daily activities, and suicidal behavior. The items are assessed based on a Likert scale, where impairments are scored from 0 to 4 points. The final score of the scale corresponds to the sum of the scores for each individual item. A null score indicates the absence of SAD-related impairment, whereas a score of 4 suggests severe impairment associated with SAD. In the original study, the scale presented good internal consistency, with a Cronbach’s alpha of .87 for the evaluation of symptoms over the previous two weeks and of .90 for symptoms in the course of life. The scale also presented a high correlation with general measures of functional impairment, suggesting its ability to assess functional impairment, as well as a high correlation with symptom severity measures (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994).
Procedures
Translation and adaptation of the DP
After the authors of the DP granted their authorization for the use of the scale, the instrument was independently translated from English into Portuguese by three mental health professionals with proper knowledge of the English language and of assessment instruments. The three versions were compared and analyzed by a fourth evaluator who, along with the first author, proceeded to a discussion on the agreement or discrepancy of items, reaching a consensus which maintained the original structure of the instrument. An interview guide was developed to support the application of the scale. The consensus version was back-translated by a bilingual psychiatrist, blind in relation to the original English version. The back-translated version was then presented to the authors of the original scale (Michael Liebowitz) for verification and confirmation of their authorization. The authors considered the present version of the scale adequate and did not present any further suggestions.
Development of the interview guide
Considering that the DP requires the judgment of health professionals, an interview guide was developed to assist and systematize the use of the scale. First, anchor-questions were created for each of the eight domains or areas of the scale, highlighting clarification and complementary questions. In general, between 8 and 10 questions specifically related to each domain were formulated. Based on the possible answers, a script of additional questions was created, aimed at offering support for the assessment of severity in each topic (around 5 questions per topic). Figure 1 illustrates a part of this script related to domains ‘school’ and ‘friendship’. The interview was always conducted in accordance with the script of guide-questions, in the same sequence and with an average duration of 12 minutes.
Figure 1. Guide questions for the ‘School’ and ‘Friendship’ domains of the Disability Profile/Clinician-Rated (DP) scale.
The script was assessed by eight mental health professionals with broad clinical experience and trained in the use of evaluation instruments who individually revised the interview guide to evaluate clarity, semantic characteristics, and appropriateness of the questions.
Subsequently, the researchers individually rated the scale in a role-playing situation in which they watched video recordings with interviews conducted using the interview guide with five subjects (three with SAD and two without SAD). Disagreements and suggestions were discussed, resulting in the final version after a minimum agreement of 80% was reached among researchers.
Data collection
First, the self-rated SAD screening instruments SPIN and MS were collectively administered in the classroom. All possible SAD cases were identified using the criteria proposed by Connor and colleagues (2001), in a total of 473 subjects. We point out that the sensitivity of MS for the diagnosis of SAD is 0.89 and specificity is 0.90.
Another 183 volunteers with no SAD indicators were also drawn from this large sample, still in accordance with Connor et al. (Reference Connor, Kobak, Churchill, Katzelnick and Davidson2001). This subsample was then contacted by telephone in order to respond to module F of the SCID-IV for the confirmation of the diagnosis and absence of SAD. Telephone interviews were conducted according to the methodology proposed by Crippa et al. (Reference Crippa, Osório, Del-Ben, Santos-Filho, Freitas and Loureiro2008). With this procedure, 178 subjects with SAD and 152 volunteers without SAD were selected, in a total of 330 participants.
In a third step, participants were randomly contacted once again for a telephone interview conducted by the same psychiatrists in order to complete additional modules of the SCID for the evaluation of the presence of other psychiatric co-morbidities. In this phase, 54% of the participants were excluded due to the following reasons: 20% were not located after three contact attempts by telephone, 25% showed no interest or availability to continue participating in the study, 7% were excluded due to the presence of co-morbidities and 2% because of incorrect completion of the study forms, amounting to a total of 173 participants. The study interviews per se were then scheduled for eligible volunteers who accepted to participate.
Participants were assessed individually in university rooms with sufficient privacy and, after signing the informed consent, were interviewed individually by four mental health professionals (two occupational therapists and two psychologists) who took turns as interviewer and observer, making up different pairs. Each evaluator applying the DP was blind in regard to the groups’ origin. Interviews were conducted following the interview guide and while the evaluators conducted the interview and scored the scale, the observer watched the interview and scored the scale independently.
Data analysis
The analysis of the demographic and clinical data was performed by applying descriptive and non-parametric statistical tests. The groups were compared with chi-squared tests. The DP scores of the SAD and non-SAD groups were compared using the Mann-Whitney test.
The levels of inter-rater reliability on the individual items of the DP were analyzed using the Kappa coefficient of agreement for two time- frame parameters: lifetime and previous two weeks.
The DP factorial structure study was carried out through the analysis of principal components aimed at identifying those elements that explained the variance of the construct evaluated. For the assessment of the internal consistency of the items of the scale, Cronbach’s alpha was used. The study of the factorial structure of the DP was carried out by Exploratory Factor analysis, using Principal Components extraction procedure and Varimax rotation, both for the lifetime and previous two weeks items. The Kaiser-Meyer-Olkin (KMO) index was calculated and Bartlett’s test of sphericity was used in order to verify the sample adequacy (N = 84).
Spearman’s correlation coefficient was used to evaluate the association between functional impairment as assessed with the DP and SAD-related symptoms related as assessed with the SPIN.
In all the statistical tests, the level of significance of p ≤ .05 was adopted.
Results
Sociodemographic characterization
The sociodemographic characterization of the sample as a function of the presence or absence of SAD is presented in Table 1. The table shows that, from the statistical standpoint, the groups do not differ in respect to the variables examined.
Table 1. Sociodemographic characteristics of the participants and statistical comparison between the SAD and non-SAD groups
Note:
(*) Chi-square test.
Psychometric properties of the DP
Regarding the descriptive analysis of the DP, the data referring to the mean, median, standard deviation, minimum and maximum scores, and skewness for SAD and non-SAD participants are presented in Table 2.
Table 2. Disability Profile/Clinician-Rated -mean, median, standard deviation and comparison between SAD and non-SAD groups
Note:
Item1- School; item2 –Work; item3 – family; item4- marriage/dating; item5- friendship; item6- other interests; item7- self-care and self-maintenance; item8- suicidal behavior; L- lifetime; 2W- last two weeks.
Statistical differences were observed across the groups for seven items in the assessment of SAD-related impairment over lifetime and in the previous two weeks.
The comparison between groups showed that, for items with statistically significant differences, values of the SAD group were higher than those of the non-SAD group.
For the lifetime parameter, a higher score was observed in those items related to romantic relationships/dating, friendship, and activities of interest. In the assessment of the previous two weeks, the highest mean score was observed in the item related to work and, similar to the lifetime assessment, romantic relationships/dating and friendship.
Regarding the inter-rater reliability of the DP, the Kappa correlation coefficient for all the items was quite satisfactory, ranging from .75 to .93 (p < .05), indicative of good reliability among evaluators.
The results concerning the internal consistency of the DP (α) are shown in Table 3.
Table 3. Internal consistency study (Cronbach’s Alpha) for Disability Profile/Clinician-Rated, for the SAD groups (n = 84), non SAD groups (n = 89) and total group (n = 176)
Note:
SAD = Social Anxiety Disorder.
For result of Bartlett’s test of sphericity for the lifetime prevalence was 89.58 (p < .001) and the KMO index was .54, which is not adequate to support the analysis and gives the data an exploratory nature. The initial factorial solution of the DP showed that two factors accounted for 51.87% of the data variance, with Factor 1 alone accounting for 35.79% of the variance.
For the prevalence in the previous two weeks, the KMO index was .73 and Bartlett’s test of sphericity was 68.06 (p < .001), moderately adequate for conducting the analysis. The initial solution of the DP showed two factors that initially accounted for 51.63% of the data variance, with 35.92% of the variance explained by Factor 1 alone.
The principal components were extracted by means of the Varimax rotation technique with Kaiser standardization (Kaiser, Reference Kaiser1974), creating a new factorial matrix that relates individual items on the scale to the extracted factors, as presented in Table 4.
Table 4. Factor analysis: factorial matrix for individual items in Disability Profile/Clinician-Rated (DP. in their lifetime and in the last two weeks for the SAD group (n = 84)
Note:
DP = Disability Profile/Clinician-Rated; * = item 2 (work) was withdrawn from the analysis of the main components considering the small number of participants who worked (7.1%).
For the lifetime prevalence, it can be seen that Factor 1 initially consisted of four items, with the “Friendship” item presenting the same factorial load in Factor 2 and being grouped with it as a function of the greater proximity of its contents to the items grouped in the factor in question. With this configuration, Factor 1 can be called ‘Performance and Relationships with Family Members’ (α = .65), and Factor 2 ‘Self-Care and Self-Maintenance Activities, Affective Relationships and Suicidal Behavior’ (α = .50). Regarding the prevalence in the previous two weeks, the same factorial structure and very close factorial loads of each item were detected. Thus, the same denomination was adopted for the two factors, which presented an internal consistency of .63 and .51, respectively.
It should be noted that, in naming these dimensions, we tried to maintain the labels used in the reference study. Despite the similarities, certain specificities were observed in the factorial distribution related to the dimensions of time. Thus, in the case of lifetime prevalence, Factor 1 included ‘Performance’ and ‘Relationships’ and Factor 2 included ‘Activities’; while in the assessment of the two-week prevalence Factor 1 grouped ‘Relationships’ and Factor 2 included ‘Performance’ and ‘Activities’.
With the aim of evaluating the association between functionality constructs and symptoms related to SAD, the correlation among factors extracted from the analysis of the principal components with the SPIN subscales was analyzed, namely, fear, avoidance, and physiological symptoms, as presented in Table 5.
Table 5. Correlation among factors extracted from the Analysis of the Disability Profile/ Clinician-Rated, in their lifetime and in the last two weeks, with the subscales from the Social Phobia Inventory (SPIN) for the SAD group (n = 84)
Note:
DP = Disability Profile/ Clinician-Rated; * = p ≤ .05; ** = p ≤ .01; SPIN = Social Phobia Inventory; Factor 1: Performance and Relationship with Family Members, Factor 2: Self-care and self-maintenance activity, Affective Relationships and Suicidal Behavior.
In the lifetime parameter, a significant correlation between the two factors extracted from the analysis of the principal components and the ‘Fear’ and ‘Avoidance’ subscales was observed. The greatest correlation was observed between Factor 2 (self-care and self-maintenance activity, affective relationships and suicidal behavior) and the ‘Fear’ subscale, with a correlation coefficient of .37.
In the two weeks parameter, correlations were observed with the three subscales (‘Fear’, ‘Avoidance’ and ‘Physiological Symptoms’). The greatest correlation was observed between Factor 1 (affective and family relationships) and the ‘Avoidance’ subscale, with a coefficient of .36.
An association was observed between fear and avoidance symptoms assessed with the SPIN and the impairment assessed by mental health professionals, related to the relationships and performance assessed with the DP for the two time parameters.
Discussion
The clinical raters identified more impairment in daily functioning in the SAD group compared to the non-SAD group of Brazilian university students. The greatest mean was observed in the item related to marriage/dating for the SAD group in the lifetime parameter, showing that this is an area of significant difficulty for carriers of this disorder, consonant with what has been described in the literature (Furmark, Reference Furmark2000; Safren et al., 1996–1997; Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994; Stein & Kean, Reference Stein and Kean2000; Wittchen & Beloch, Reference Wittchen and Beloch1996). This result is very similar to reports by a number of authors that subjects with SAD tend to remain single and have fewer romantic relationships throughout their lifetime (Mendlowitz & Stein, Reference Mendlowicz and Stein2000).
In the reliability study of the DP, good agreement levels were found among raters. Nevertheless, as far as we are concerned, there are no comparison parameters with other studies since no other inter-rater reliability studies of the DP were identified. It can be considered that the interview guide contributed to reduce the variability of information obtained in reliability studies, as demonstrated by other groups (Crippa, Sanches, Hallak, Loureiro, & Zuardi, Reference Crippa, Sanches, Hallak, Loureiro and Zuardi2001; Osório et al, Reference Osório, Crippa and Loureiro2006).
Regarding the internal consistency -an additional indicator of reliability- the values obtained in the present study might be deemed acceptable, suggesting a moderate internal consistency, despite the lower values found here compared to the original study (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994) and to another previous investigation (Hambrick, Turk, Heimberg, Schneier, & Liebowitz, Reference Hambrick, Turk, Heimberg, Schneier and Liebowitz2004).
The factors identified in principal components analysis for the two temporal parameters of the DP were found to contain the indicators related to the difficulties and suffering experienced in relationships and in the participation in necessary activities and activities of interest. These indicators have been reported in studies on subjects with SAD based on the evaluation of quality of life (Mendlowicz & Stein, Reference Mendlowicz and Stein2000) and in studies evaluating damages in daily activities (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994; Stein & Kean, Reference Stein and Kean2000; Wittchen & Fehm, Reference Wittchen and Fehm2003. These findings agree with the frequent division of SAD symptoms into interaction and performance symptoms. The desire to live variable seems to be correlated with situations of difficulties in relationships and in the participation in daily activities.
We wish to point out that the reduced number of factors detected in the study in question may be related to the fact that the scale, as proposed by the authors of the original study (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994), is a concise instrument focusing on a minimum set of variables indicating functional damages related to SAD.
Consistent with the original DP study (Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994), we also verified the association between functional impairments and SAD symptoms. We detected a relationship between fear and avoidance symptoms related to SAD and impairments in everyday life. It should be highlighted that the moderate correlations with the SPIN subscales suggest that the specificities of functional impairments require specific instruments for a better evaluation.
In the present study, as in previous reports (Hambrick et al., Reference Hambrick, Turk, Heimberg, Schneier and Liebowitz2004; Schneier et al.,Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994), the functional impairments showed good, moderate and positive correlations (Pereira, 2004) with SAD symptoms, suggesting that the more severe the SAD symptoms, the greater the impairment in activities that support daily routines, and vice-versa.
Regarding the discriminative validity of the DP, the scale was found to be sensitive to differentiate impairments in SAD and non-SAD cases, considering that the assessment of the SAD group yielded higher mean scores, that is, greater functional impairment.
The DP presented good psychometric properties for the Brazilian population of university students with and without SAD, considering the internal consistency indicators and the analysis of the principal components in comparison with studies involving samples at different age ranges and cultural contexts (Hambrick et al., Reference Hambrick, Turk, Heimberg, Schneier and Liebowitz2004; Schneier et al., Reference Schneier, Heckelman, Garfinkel, Campeas, Fallon, Gitow and Liebowitz1994).
The relevant contribution of this instrument for the systematic assessment of functional impairments deserves to be highlighted, both in comparison studies between groups regarding the presence of SAD symptoms and in intervention studies as well.
As limitations of the study, the elevated number of drop-outs and missing subjects constituted the main obstacle. This could be a result of the fact that SAD consists of everyday life impairments that are also expressed in terms of avoidance and non-participation in surveys or studies. Another limitation was the specificity of the sample, which consisted exclusively of university students with ages close to the onset of SAD, normally centered in the adolescence period and beginning of adult life, which may have contributed to the inclusion of people with less impairment or who may not have associated experienced impairments with SAD yet.
The DP is an instrument with adequate psychometric properties that can be used for the assessment of university students and constitutes a useful tool in mental health practices targeted to improve functional outcomes. Future studies in different cultures, samples and contexts are still desirable and necessary.
