One important aim of psychological research is to establish whether a construct and its measures are pertinent across culturally diverse groups (Byrne & Campbell, Reference Byrne and Campbell1999). The measurement field of emotional intelligence (EI) is no exception. Nowadays, the widespread application of assessment methods to different languages and cultural backgrounds (Batista-Foguet, Saris, Boyatzis, Guillén, & Serlavós, Reference Batista-Foguet, Saris, Boyatzis, Guillén and Serlavós2009) raises issues of cultural comparability and cross-cultural meaning, thus challenging the validity and the interpretation of EI measures (Emmerling & Boyatzis, Reference Emmerling and Boyatzis2012).
Measurement invariance is related to the degree to which items and constructs have an equivalent meaning for individuals of different cultural backgrounds (Little, Reference Little1997). It occurs when items and its answering scales are perceived and understood correspondingly across diverse samples, assuring that a given instrument is assessing similarly the same psychological construct across different groups (Elosua & Hermosilla, Reference Elosua and Hermosilla2013).
The issue of measurement invariance can be particularly dangerous, especially in the field of EI. As a consequence of non-invariant measures, relevant conclusions concerning the generalization of the EI concept across different cultural groups could be at stake. The fact that those abilities that allow one to express, perceive, understand, and manage emotions could not be generalized across different cultural groups, appears to be a central matter, both in the assessment and conceptualization of this construct (Ghorbani, Davison, Bing, Watson, & Mack, Reference Ghorbani, Davison, Bing, Watson and Mack2002).
In the growing field of EI, although a few cross-cultural studies have been described, the measurement invariance of EI’s measures are now starting to be addressed (Ekerman, Saklofske, Austin, & Stough, Reference Ekerman, Saklofske, Austin and Stough2011; Ghorbani et al., Reference Ghorbani, Davison, Bing, Watson and Mack2002; Karim & Weisz, Reference Karim and Weisz2010; Li, Saklofske, Bowden, Yan, & Fung, Reference Li, Saklofske, Bowden, Yan and Fung2012; Parker et al., Reference Parker, Saklofske, Shaughnessy, Huang, Wood and Eastabrook2005). In particular, based on Mayer and Salovey’s ability model (Reference Mayer, Salovey, Salovey and Sluyter1997) one of the widely used self-report EI instruments is the Emotional Skills and Competence Questionnaire (ESCQ; Takšić, Mohorić, & Duran, Reference Takšić, Mohorić and Duran2009). This measure assesses individual’s self-perceptions about EI, but beyond its extensive use lacks cross-cultural measurement invariance studies (i.e. Portugal, Finland, Sweden, Slovene, Spain, Japan, Italy, China, India, Argentina, United States, Iran, and Austria; Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Takšić et al., Reference Takšić, Mohorić and Duran2009).
For this reason, this study is intended to bring awareness to the validity of cross-cultural studies in the emotional abilities field, since it is crucial that more empirical evidence support the replicability of EI and emotional-related constructs as identical sets of psychological constructs in other languages and cultures.
Measurement Invariance on EI Research
Following an initial publication, EI has witnessed increasing development during recent decades (Mayer & Salovey, Reference Mayer, Salovey, Salovey and Sluyter1997). Researchers have been exploring EI as an individual-differentiating construct in several settings and cultures.
Currently, considering either a theoretical or an evaluative perspective, the literature presents two distinct views of EI: ability and trait models. The ability model perceives EI as a capacity to engage in valued behavior and involves a certain degree of mutability (Mayer, Caruso, & Salovey, Reference Mayer, Caruso and Salovey1999), referring to ‘an individual’s ability to perceive accurately, appraise, and express emotion; access and/or generate feelings when they facilitate thought; understand emotions and emotional knowledge; and to regulate emotions to promote emotional and intellectual growth’ (Mayer & Salovey, Reference Mayer, Salovey, Salovey and Sluyter1997, p. 5). Trait EI model considers EI as ‘a constellation of behavioral dispositions and self-perceptions concerning one’s ability to recognize, process, and use emotion-laden information’ (Petrides & Furnham, Reference Petrides and Furnham2003, p. 278). Anchored in the diverse EI’s theoretical conceptualizations available in the literature (Zeidner, Roberts, & Matthews, Reference Zeidner, Roberts and Matthews2002) several measures to assess EI have been established (Mayer & Salovey, Reference Mayer, Salovey, Salovey and Sluyter1997). The measurement of EI can generally be differentiated into two major types: self-report and performance measures (Ciarrochi, Chan, Caputi, & Roberts, Reference Ciarrochi, Chan, Caputi, Roberts, Ciarrochi, Forgas and Mayer2001). Although EI’s assessment is still in its early infancy, the proliferation of measures within EI’s different conceptualizations lends urgency to the search of consensus in the measure’s validity. Research has pointed out that self-report measures may have lower validity due to its dependence on individual’s opinion and understanding, which may be affected by vulnerability to social desirability factors (Roberts, Zeidner, & Matthews, Reference Roberts, Zeidner and Matthews2001). However, EI’s performance assessment also presents problems in determining correct answers to emotional content stimulus. Therefore, it becomes difficult to apply the exact criteria of scoring to the tasks (Roberts et al., Reference Roberts, Zeidner and Matthews2001).
Moreover, some authors consider that the EI ability-based perspective (Mayer, Caruso, & Salovey, Reference Mayer, Caruso and Salovey1999), similar to any other measure of intelligence, should assess the individual’s actual capacity to perform through objective performance measures (Ciarrochi et al., Reference Ciarrochi, Chan, Caputi, Roberts, Ciarrochi, Forgas and Mayer2001) and ability testing. Instead, self-report measures should be exclusive of wider perspectives of EI that include personality aspects and attributes (i.e., Boyatzis, Goleman, & Rhee, Reference Boyatzis, Goleman, Rhee, Bar-On and Parker2000), since these measures only appraise individuals’ own beliefs about their capacities (Ciarrochi et al., Reference Ciarrochi, Chan, Caputi, Roberts, Ciarrochi, Forgas and Mayer2001). However, several performance and self-report measures exist in the literature and should support further development of specific research comparing the validity of both types of EI’s assessment.
The fact that self-report measures involve less costs and resources and allow data gathering to be more efficient (Li et al., Reference Li, Saklofske, Bowden, Yan and Fung2012) facilitated its generalized use in several contexts and, more recently, in cross-cultural studies of EI. In fact, nowadays, a great amount of EI instruments are being developed and the generalized use of non-assessed invariance measures could influence the meaningfulness and generalization of results. The first measurement invariance level - configural invariance - provides support for the equivalence of the form of the factor model across different groups, implying a similar conceptualization of what EI represents to the different cultural groups. The metric invariance implies that the regression of the indicator variable on the latent trait has parallel slopes across groups for all indicator variables/items. Further, within the frame of metric non-invariance, the items of a scale might not represent or have the same weight in the respective dimension in the different groups. The scalar invariance is a test of the hypothesis that the vector of item intercepts is invariant across groups. This level of measurement equivalence can be obtained when two metric measures have the same measurement unit and the same origin (van de Vijver & Tanzer, Reference van de Vijver and Tanzer2004). According to van de Vijver and Tanzer (Reference van de Vijver and Tanzer2004), this particular strong measure of invariance assumes a completely bias-free measurement. The bias can complicate the achievement of higher levels of invariance. In fact, the authors contended that the method and item bias can jeopardize scalar equivalence. For instance, an item that consistently favors one cultural group will conceal the underlying real cross-cultural differences in the scores on the construct that is being assessed. Therefore, if scalar non-invariance is confirmed, it means that although the individuals might have the same level in the EI domain, participants from the different cultural groups still tend to score higher or lower on the different items. Such an outcome misrepresents the impression that one of the groups exceeds the other, which will lead to incorrect conclusions.
The task of examining the validity and robustness of EI measures across different cultural groups is particularly important, since it is acknowledged that the learning, the control, the expression, and the perception of emotion can be socially and culturally molded, maintained or even influenced by specific social standards (Parker et al., Reference Parker, Saklofske, Shaughnessy, Huang, Wood and Eastabrook2005). EI is likely to vary across cultures, as it is influenced by one’s attitudes and beliefs within a specific social and cultural context. In fact, research suggests that there may be significant cultural variability in particular skills associated with the main EI models (Emmerling & Boyatzis, Reference Emmerling and Boyatzis2012). Therefore, the processes underlying the EI factors and their manifestations across cultures may differ as a consequence of cultural roles in the interpretation and development of emotions (Karim & Weisz, Reference Karim and Weisz2010).
However, several studies that address the comparability of emotion-related domains have not tested instruments’ measurement invariance. For instance, the classical conclusions of Matsumoto (Reference Matsumoto1993) that compare American undergraduate students with different cultural backgrounds found ethnic group differences in emotions judgments, rules display, and even in their emotional expressions’ perception. At some extent, this conclusion could be misleading without certainty of measurement invariance. Also, individualism-collectivism cross-cultural research has evidenced differences across a wide range of emotion-related abilities that comprise the construct of EI: individualistic cultures seem to have better skills at recognizing, understanding (Matsumoto, Reference Matsumoto1989), expressing (Fernandez, Carrera, Sanchez, Paez, & Candia, Reference Fernandez, Carrera, Sanchez, Paez and Candia2000), and regulating their emotions (Gross & John, Reference Gross and John2003) than collectivistic cultures. In this sense, countries with different individualistic-collectivistic backgrounds can face difficulties accounting for their instrument’s measurement invariance. In fact, without checking for measurement invariance it is not clear if differences in EI scores in countries with different cultural backgrounds are due to differences in the interpretation of EI items’ scales or indeed reflect real cultural differences in EI domains. In this case, it is extremely relevant that upcoming studies continue to address the validity of cross-cultural comparisons.
In the EI field, the invariance of its measures is now starting to be addressed (Ekerman et al., Reference Ekerman, Saklofske, Austin and Stough2011; Ghorbani et al., Reference Ghorbani, Davison, Bing, Watson and Mack2002; Karim & Weisz, Reference Karim and Weisz2010; Li et al., Reference Li, Saklofske, Bowden, Yan and Fung2012; Parker et al., Reference Parker, Saklofske, Shaughnessy, Huang, Wood and Eastabrook2005). Ghorbani et al. (Reference Ghorbani, Davison, Bing, Watson and Mack2002) tested for Trait Meta-Mood Scale (TMMS) invariance among Iranian and U.S. university students and, actually, it was the only study to prove full measurement invariance of an EI measure. Nonetheless, studies using various EI measures have achieved different levels of invariance. For instance, Parker et al. (Reference Parker, Saklofske, Shaughnessy, Huang, Wood and Eastabrook2005) and Siu (Reference Siu2009) reported configural invariance for the Bar On EQ-i Youth version and Emotional Intelligence Scale, respectively, but failed to prove metric invariance. More recently, Karim and Weisz (Reference Karim and Weisz2010) confirmed configural, metric, and partial scalar invariance when comparing French and Pakistani Mayer-Salovey-Caruso Emotional Intelligence Test (MSCEIT) versions. In Ekerman and colleagues (Reference Ekerman, Saklofske, Austin and Stough2011) and Li and collaborators (Reference Li, Saklofske, Bowden, Yan and Fung2012)’s studies, the results confirmed the configural and metric invariance of EQ-i in four cultural contexts and of the Wong and Law Emotional Intelligence Scale (WEILS) in three Chinese student groups, respectively.
In particular, the ESCQ (Takšić et al., Reference Takšić, Mohorić and Duran2009) is comprised of three dimensions — express and label emotion, perceive and understand emotion, and manage and regulate emotion. The instrument was originally developed in Croatia, but it has been widely diffused into other countries (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Takšić et al., Reference Takšić, Mohorić and Duran2009). In general, the ESCQ has shown good psychometric properties across different studies, confirming the three-factor structure underlying the ESCQ dimensions, and revealing good reliability (between .72 and .92), positive correlations between the dimensions (between .49 and .54) (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Takšić et al., Reference Takšić, Mohorić and Duran2009), and presenting absolute and relative modest fit indices (GFI = .87; AGFI = .86; RMSEA = .055; Takšić et al., Reference Takšić, Mohorić and Duran2009; NFI = .93; CFI = .94; RMR = .04; RMSEA = .04; Stocker & Faria, Reference Stocker and Faria2012). Despite the ESCQ’s proliferation in an international context and the existence of several studies in different countries that confirm the validity of its factor structure (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Stocker & Faria, Reference Stocker and Faria2012; Takšić et al., Reference Takšić, Mohorić and Duran2009), this instrument still needs to deepen its cross-cultural validity.
In a recent study, Molander, Holmström, and Takšić (Reference Molander, Holmström and Takšić2011) explored the differential item functioning (DIF) in the ESCQ’s scale across several countries (i.e. Balkan (Croatia, Serbia and Slovenia) and Nordic countries (Finland and Sweden). In general, the results highlighted the category “Traditional (related to societies where religion and religious values, traditional family and nationalistic values are considered important) vs. Secular-rational” (features societies which emphasize well-being, self-expression, and quality of life over economic and physical security) of the World Value Survey (WVS) Footnote 1 , as the better indicator for DIF differences across countries, beyond language features. In particular, several items of Perceive and Understand Emotion and Express and Label Emotion’s dimensions had higher DIF values across the countries, thus being more sensible to cultural differences. The results evidenced the cultural effects in the interpretation of the items’ meanings among the countries, even considering the errors in the test translation from the Croatian original and the methodological errors (Molander et al., Reference Molander, Holmström and Takšić2011). As stated in Molander et al. (Reference Molander, Holmström and Takšić2011)’s study, the lack of a supporting theory in the area of EI that allows the prediction of variables cross-cultural variation, make it difficult to reveal the enlightenment of EI’s pattern differences between countries and cultures.
In the present study, two European countries where ESCQ use was widespread were selected: Croatia, where it was originally developed, and Portugal, where studies in different settings took place (Costa & Faria, Reference Costa and Faria2014, Reference Costa and Faria2015; Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Stocker & Faria, Reference Stocker and Faria2012). Since it pertains to different cultural backgrounds, this study intents to contribute by enlightening the European cross-cultural emotion abilities — Croatia is a southeastern European country that achieved relatively recent independence in the early 1990s. It is currently facing entry into the EU and is characterized as being more open to the western world. Conversely, Portugal is a western European country that moved from dictatorship to democracy in the early 1970s and had an ulterior debut in the EU. Due to its background in cultural differences, it is possible that Portugal can have more individualistic and secular features when compared to the Croatian setting, mainly because of the shorter period of time during which Portugal was subjected to dictatorial restrictions.
Moreover, since measurement invariance has never been compared within the Portuguese and Croatian contexts, the present research will highlight possible differences in the way the two cultures interpret the items/scale based in their different cultural backgrounds. As found in previous studies, ESCQ original Croatian version presented DIF items when compared with different cultural countries (i.e. Non-Balkan countries; Molander et al., Reference Molander, Holmström and Takšić2011). Similarly it is expected that the Croatian and Portuguese versions of this measure, underlying cultural countries differences (eastern/western countries; individualism/collectivism features), might reveal dissimilarities in the interpretation of EI items and answering scales.
Therefore, in the present study, it is hypothesized that the Portuguese and Croatian versions of ESCQ will confirm the configural invariance (i.e., the factorial validity of the ESCQ measure) supported by the extent of recent cross-cultural research (Molander, Holmström, & Takšić, Reference Molander, Holmström and Takšić2009; Takšić et al., Reference Takšić, Mohorić and Duran2009) and achieve partial metric and scalar invariance due to non-invariant items between the two versions.
These results will allow researchers to gather insight about the way in which different aspects of EI (expression, perception, understanding, and managing of emotions) are operationalized by the ESCQ across cultures. In particular, this research will add evidence for the precursor field of EI’s measurement invariance by presenting a ESCQ cross-cultural measurement invariance study in two European countries.
Method
Sample and Procedure
Portuguese Sample. 627 secondary school students provided data suitable for the analysis. The Portuguese sample comprised 52.6% of female students and 47.4% of male students, with ages ranging from 14 to 21 years old (mean age = 15.5; SD = .76) and attending several secondary courses.
Croatian Sample
562 secondary students participated in this study. This sample consisted of 67.2% females and 32.4% males, with ages ranging from 14 to 19 years old (mean age = 16.3; SD = 1.07) and attending several secondary courses.
The ESCQ scale was answered individually by each participant in both countries, in classrooms’ collective administrations, during school time and in the presence of a researcher and the teacher. The aims of the study, the confidentiality and the anonymity were explained and guaranteed to all the participants.
Instrument
The Emotional Skills and Competence Questionnaire (ESCQ; Takšić et al., Reference Takšić, Mohorić and Duran2009) is a self-report EI measure using a 5-point Likert type scale (1 = never to 5 = always). Originally developed in the Croatian context with 45 items, within Mayer and Salovey’s (Reference Mayer, Salovey, Salovey and Sluyter1997) theoretical framework, this measure has been adapted and validated to several cultural contexts (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Takšić et al., Reference Takšić, Mohorić and Duran2009). The results of the studies with the original ESCQ version confirmed three subscales with decent reliability: between .81 and .90 alpha values for Perception and Understanding emotions, .78 and .88 for Express and Label emotions, and .67 and .78 for Manage and Regulate emotions. Positive and moderate correlations between the subscales were achieved (.35 to .51), which justify the linear combination measure of overall emotional competence with a reliability between .88 and .92 alpha values (Takšić, Reference Takšić, Lackovic-Grgin and Penezic2001). Moreover, confirmatory factor analysis with a high-school students sample (N = 1460), confirmed the three-factor structure and revealed acceptable goodness of fit indices (GFI = .87; AGFI = .86; RMSEA = .055; Takšić et al., Reference Takšić, Mohorić and Duran2009).
The Portuguese version of ESCQ was translated based on the double or back-translation method of the English version of the instrument. It is considered as the most reliable method of attaining semantic equivalence between the source language and the target language (Duffy, Reference Duffy2006), involve at first the translation of the instrument from the source language into the target language (forward translation) by one bilingual expert and then the independent translation of two experts of the forward-translated version into the source language (back translation or double translation) without previous knowledge of the original instrument (Duffy, Reference Duffy2006). To come up with a final version of the instrument, whereas the translation is both conceptually and linguistically equivalent, the researcher compares the expert’s versions and discuss with them the modifications and adjustments required. The available English version of the ESCQ was also attained based on the double or back-translation method of the instrument’s source Croatian version.
The ESCQ Likert-type scale has 5 points (1 = never to 5 = always) and it is based in a frequency scale that relates to the prevalence of the behavior or thought (Brown, Reference Brown2004), in this case to the frequency of emotional abilities. Research has been pointing out that a frequency scale provides a more accurate indication of people’s behaviors than do reports on the intensity of beliefs, that relate to perceived importance of values or beliefs (Brown, Reference Brown2004) and, therefore, less problematic in cross-cultural comparisons where individuals from different cultures may interpret the agreement scales differently (Shulruf et al., Reference Shulruf, Alesi, Ciochină, Faria, Hattie, Hong and Watkins2011).
Although the original scale had 45 items, in this study, a 42-items ESCQ version was used for both cultural groups due to several items’ refinements in previous psychometrics studies (Stocker & Faria, Reference Stocker and Faria2012). It comprises 3 subscales: Perceive and Understand Emotion (PUE; 14 items – “I am able to tell the difference if my friend is sad or disappointed”), Express and Label Emotion (EE; 14 items – “I can easily name most of my feelings”), and Manage and Regulate Emotion (MRE; 14 items – “I can maintain a good mood, even when the people around me are in a bad mood”). All of the items of the two versions of the instrument were presented in the direct form, so that could not influence the response style of the participants.
Data analysis
Measurement Invariance
The invariance analyses consisting of two steps were performed with Multiple-Group Confirmatory Factor Analyses. A sequence of increasingly restrictive CFA models was examined to evaluate the measurement invariance of the ESCQ scale across both contexts (Portugal vs. Croatia).
Initially, CFA was used to test the goodness-of-fit of the baseline model in each cultural sample. The next step of the invariance routine consisted of comparing the fit of more constrained models with the baseline model successively, using multi-group analyses, which assessed the fit of the three-factor model simultaneously across Portuguese and Croatian samples. In order to assure the cross-validity of measurement invariance results, the main sample was previously split in two random halves, one used to achieved and test the baseline model in each cultural context and the other to perform the testing of more restrictive models in measurement invariance’s analyses. Preliminary descriptive data analyses and ESCQ’s reliability for the several groups using Cronbach’s alpha coefficient (SPSS 21.0) were conducted.
The basic level of measurement invariance tested is configural invariance. The central requirement is that the same item must be an indicator of the same latent factor in each group. If the multi-group model with no cross-sample constraints imposed (Baseline Model; Model 1) presented appropriated fit indices, this level of invariance is achieved, which supports that similar latent variables are present in the groups (Byrne, Reference Byrne2004).
The following invariance levels - metric and scalar – are met when the subsequent constrained models confirm to be as good as the baseline model: there are no significant losses of fit in the more restrict model when comparing to the baseline/unconstrained model [χ2, RMSEA and CFI differences in fit; i.e. non-significant χ2 dif = χ2 unconstrained model - χ2 constrained (metric or scalar) model] and also the restrict model provides appropriated fit indices of the multi-group model.
The level of metric invariance represents the strength of the linear relation between each factor and its associated items. Given the appropriated fit indices of the multi-group model (with all factor loadings constrained to be equal across samples – Model 2) and no significant loss of fit comparing to the baseline model this level of invariance is met, confirming that the loading of each item on the underlying factor is equal in two (or more) groups.
The third level of invariance is the scalar invariance: in testing this form of invariance, intercepts of the measured variables are constrained to be equal across groups, in addition to factor loadings of the latent variables (Model 3). This level of invariance is confirmed by finding a good fit of the model to the data, and scores from different groups have the same unit of measurement (factor loadings) as well as the same origin (intercept) (Byrne, Reference Byrne2004).
The data analysis was conducted using AMOS (v. 21, SPSS Inc.) with maximum likelihood estimation. Maximum likelihood estimation method requires the assumption of the multivariate normal distribution of continuous variables. This estimation method is relatively robust to violation of these conditions, as the case of ESCQ ordinal variables in this study, if the number of grade response categories is large enough (5 or more; Bollen & Barb, Reference Bollen and Barb1981; Dolan, Reference Dolan1994) and univariate skewness and kurtosis are not large (range between –1 and 1; Bollen, Reference Bollen1989; Muthén & Kaplan, Reference Muthén and Kaplan1985). In this study the ESCQ variables are ordinal but comprise a 5-points scale grade response. Moreover, the exploration of the ESCQ variables distribution shown that skewness and kurtosis are not large enough (range between –.955 and .470, and between –.987 and .274, respectively), therefore, confirming the adequacy of ML estimation method in this study.
Grounded on absolute and relative multiple criteria fit indices, Hu and Bentler’s (Reference Hu and Bentler1999) cutoff criteria for goodness-of-fit indices were applied. The absolute misfit index used was the root mean square error of approximation (RMSEA). The relative goodness-of-fit indexes were the comparative fit index (CFI) and the Tucker-Lewis Index (TLI, or Non-Normed Fit Index: NNFI). Values of RMSEA < .08, and CFI and TLI > .90 are indicative of an acceptable fit (Hu & Bentler Reference Hu and Bentler1999). The chi-square difference (χ2 diff) test together with changes in the RMSEA, CFI (Chen, Reference Chen2007) were used to compare the more stringent models (Models 2 and 3) with the less restricted model (Model 1). Chen (Reference Chen2007) argued that changes in CFI of –.010 or less supplemented by a change of .015 or more in RMSEA would indicate non-invariance of the models.
Results
Baseline Confirmatory Factor Analysis
As prior research has provided empirical evidence for the theoretical three-factor model underlying the ESCQ (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006), our first analyses re-examined the three-factor structure of the ESCQ in the different countries samples separately. This preliminary analysis to the model’s goodness-of-fit, adjusted in separated samples, revealed no satisfactory values (χ2 (df) > 2; CFI & TLI < .80 cf. Table 1). According to the recommended practice for the identification of an acceptable CFA model (Kline, Reference Kline2010), the inspection of the parameter estimates and the examination of the modification indices were then performed.
Table 1. Fit Indices for CFA model in each sample for the original and reduced scales
Note: RMSEA = root mean squared error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index or non-normed fit index; *p < .01 for the χ2 test.
For both samples, the analyses of the estimates’ parameters revealed that although significant, some factor loadings were < .30 which lead to the items’ exclusion on ESCQ’s subscales (for instance, for both samples the items 5, 14, 23, 29, 35, 41 from ELE, and items 6, 15, 18, 21, 24, 33, 36, 42 from MRE) and others > .30 but whose exclusion would improve greatly the model goodness-of-fit (the items 1, 28, 31 from PUE, the item 20 from ELE, and item 9 from MRE). Those were the problematic items in both samples. The refinements on the CFA models following the modification indices criterion for the different samples endorsed excluded the item 25 (PUE) in the Portuguese scale for low factor loading, and the item 4, 37 (PUE), 32 and 38 (ELE) in the Portuguese scale and the item 22 (PUE) and 2 (ELE) in the Croatian scale for displaying errors’ correlation with items of others dimension’s scales and provided the higher significant change, thus reducing the discrepancy between the two models. Suitable model’s goodness of fit parameters was achieved for the three-factor model underlying ESCQ for each scale (cf. Table 1). While the final Portuguese CFA model sustained a total of 18 items (8 items in PUE, 5 in ELE and 5 in MRE), the Croatian CFA model displayed 21 items (10 items in PUE, 6 in ELE and 5 in MRE). The final model comprised 16 items overlapping between Croatian and Portuguese versions (7, 10, 13, 16, 19, 34, 40 for PUE, 8, 11, 17, 26 for EE, 3, 12, 27, 30, 39 for MRE).
As shown in table 2, the final factor loadings in the CFA models adjusted to each sample ranged from .455 to .759 for the Portuguese sample, and from .335 to .858 for the Croatian sample, which are within the recommended parameters. Moreover, correlations between the 3 factors revealed moderate associations as expected, although the Portuguese sample has displayed lower values than the Croatian one (cf. table 3).
Table 2. Standardized Factor Loadings in each sample
Note: The factor loadings values in the items 4, 22, 25, 37, 2, 32, 38 that were not displayed in the table, were not included in the final baseline country model.
Table 3. Factor Correlations for CFA model in each sample
** p < .01
Descriptive and Reliability Analysis
The means scores and standard deviations for PUE, ELE, MRE and the total ESCQ scale for the baseline models for both cultural groups are reported in Table 4.
Table 4. Descriptive statistics and reliabilities of ESCQ in both groups
Note: M = Mean; SD = Standard Deviation; CI = Confidence Interval; PUE = Perceive and Understand Emotion; ELE = Express and Label Emotion; MRE = Manage and Regulate Emotion.
The internal consistency reliability for each subscale and the total ESCQ across samples was calculated for both the original and reduced ESCQ version by the Cronbach’s alpha coefficient, with coefficients ranging from .87 to .88 for PUE, .84 to .86 for ELE, .72 to .75 for MRE, and .91 for the ESCQ total original scales and from .80 to .85 for PUE, .82 to .87 for ELE, .68 to .70 for MRE, and from .83 to .88 for the ESCQ total reduced scales, confirming in general the results of previous studies (reliability above .70; Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; cf. Table 5).
Table 5. ESCQ’s original and reduced scales reliabilities in both groups
Note: α = Cronbach’s alpha coefficient. PUE = Perceive and Understand Emotion; ELE = Express and Label Emotion; MRE = Manage and Regulate Emotion.
Moreover, the correlations between the original and reduced scales were calculated, confirming the strong correlations for all the dimensions and total scale (from .84 to .98 for the Croatian scale and from .84 to .96 for the Portuguese scale (cf. Table 6).
Table 6. Correlations between the original and reduced scale scores for the ESCQ’s dimensions and total scale
Note: PUE = Perceive and Understand Emotion; ELE = Express and Label Emotion; MRE = Manage and Regulate Emotion; ESCQ total = Emotional Skills and Competence Questionnaire total scale; **p < .01.
Testing Measurement Invariance
Configural Invariance
The configural invariance model imposes the same theoretical factor structure across samples. This means that the same indicators (i.e. 16 items overlapping between Croatian and Portuguese versions) measure the same factors in the different cultural contexts. As the multi-group model presented good fit to the data (cf. Model 1, Table 7), configural invariance across different cultural contexts was supported.
Table 7. Testing of measurement invariance of the ESCQ across countries
Note: RMSEA = root mean squared error of approximation; CFI = comparative fit index; TLI = Tucker-Lewis index or non-normed fit index; P = Portugal; C = Croatia; **p < .01 for the χ2 test.
Metric and Scalar Invariance
As the configural invariance model does not impose any measurement parameters to be identical across groups, it serves as a baseline model to evaluate subsequent more restrictive invariance models. Once we considered the baseline model as being supported, both theoretically and empirically, we then compared the baseline model with a highly restrictive metric and scalar invariance model that imposes invariant factor loadings and an invariant intercept structure respectively across different groups. Thus, having established configural invariance, metric and scalar invariance were then examined across Portuguese and Croatian groups (cf. Table 7).
Compared with the baseline model, estimated simultaneously in both groups (cf. Model 1 in Table 7), the introduction of invariance restrictions on all the factor loadings conducted to significant loss of fit in the χ2 (p < .001) and ΔCFI = .02, although ΔRMSEA = .005 was in the limit of appropriated fit (cf. Model 2 in Table 7). The inspection of the modification indices revealed that three items measuring “PUE” (Item 19 “When I meet an acquaintance, I immediately notice his/her mood”, item 34 “I am able to tell the difference if my friend is sad or disappointed” and item 40 “I do not have difficulty to notice when somebody feels helpless”) exhibited non-invariance factor loadings. As a result, it was released the restrictions on the three items’ parameters in PUE.
This partial metric invariance model (cf. Model 2.1 in Table 7), constituted by a total of 13 items (4 PUE, 4 ELE and 5 MRE items) showed an improved model fit (Δχ2 = 63.977, Δdf = 13, p = < .001, ΔCFI = .02, ΔRMSEA = –.005 (full metric invariance model) vs. Δχ2 = 9.102, Δdf = 10, p = .522, ΔCFI = –.01, ΔRMSEA = .002 (partial metric invariance model). Therefore, the partial metric invariance model (cf. Table 7) seemed to be an adequate model indicating what can be the most critical sources of non-invariance in measurement parameters of ESCQ across Portugal and Croatia.
Nonetheless, the statistical comparison between the 13 items partial scalar invariance model (the non-invariant factor loadings obtained when testing for metric invariance were set free in this model; cf. Model 3 in Table 7) and the baseline model reveals that the difference in χ2 is highly significant (p < .001; cf. Table 7), leading to the conclusion that this scalar invariance model is overly restrictive and should therefore be rejected. In order, to explore if there were scalar invariant items in the ESCQ scale, further analyses were conducted. The inspection of the modification indices revealed that two items measuring “PUE” (Item 7 “I notice when somebody feels down” and item 10 “I notice when somebody’s behavior varies considerably from his/her mood”), two items measuring “ELE” (Item 11 “I am able to express my emotions well” and item 26 “I can recognize most of my feelings”), and two items measuring “MRE” (Item 3 “I try to keep up a good mood”, item 12 “I try to control unpleasant emotions, and strengthen positive ones” and item 27 “I can maintain a good mood, even when the people around me are in a bad mood”), did not hold for scalar invariance. The model with the non-invariant items unconstrained (cf. Model 3.1 in Table 7) achieved partial scalar invariance (Δχ2 = 15.290, Δdf = 21, p = .083; ΔCFI = –.001, ΔRMSEA = .006; cf. Table 7).
Discussion
The use of imported measures requires that different levels of equivalence should be established in order to achieve both the precision and the meaningfulness of comparisons in cross-cultural research (Byrne & Campbell, Reference Byrne and Campbell1999). Although the questions related to the instrumental invariance are frequently overlooked, the present study aimed to contribute to the ESCQ’s measurement invariance, by providing empirical evidence across two European countries (Portugal vs. Croatia).
Adding empirical support to the previous research on ESCQ properties (Faria et al., Reference Faria, Lima Santos, Takšić, Räty, Molander, Holmström and Toyota2006; Takšić et al., Reference Takšić, Mohorić and Duran2009), this study explored the psychometric validity of this self-report instrument. In fact, the CFA analyses lead to a reduced ESCQ scale that confirmed the three-factor model underlying the ESCQ. Although the final version of ESCQ excluded several items, reliability and convergence analyses were conducted to assure that content validity was not compromised in the instrument’s reduced version. ESCQ’s final scale presented lower reliability compared to the original one, which can be due to reliability’s dependency of the number of assessed items (Brown, Reference Brown2001). In this case, fewer items in each dimension contributed to a decrease in ESCQ’s reliability. The Cronbach’s alpha values in the reduced ESCQ scale were, in general, appropriated (above .80) with the exception of Manage and Regulate dimension (α = .70 in the Portuguese scale and α = .67 in the Croatian scale). This particular dimension had the lower levels of reliability in the ESCQ’s original version (from .72 to .75). The EI literature has been proving the difficulty in assessing this particular complex domain, leading to results that often lack reliability consistency (Rivers, Brackett, & Salovey, Reference Rivers, Brackett, Salovey, Boyle, Matthews and Saklofske2008). Thus, the lower reliability in the reduced dimension scale of this study could perhaps be justified, not only by the exclusion of specific items, but also because it relates to more extended problems with the general assessment of this dimension in the EI domain. Moreover, how well the reduced ESCQ scale and its dimensions cover the original ESCQ’s three underlying dimensions was explored. The correlation coefficients between the original and reduced scale dimensions revealed that they are highly inter-correlated (from .84 to .97), which contributes to assure the strong similarity between the two scales and the content validity of the reduced scale. The goodness-of-fit model higher parameters and the appropriate reliability and convergence analyses’ results supported the content validity of the reduced ESCQ scale and its dimensions in both groups, thus allowing the measurement invariance analyses.
Furthermore, concerning the measurement invariance analyses, the results of the multi-group CFA indicated that the ESCQ represented the same three EI factors in each of the examined groups. The cross-cultural comparison of the ESCQ’s factorial structure supported the configural invariance and the comparability of the different facets of EI (i.e., expression, perception, and understanding and managing emotions) across the different cultures).
Additionally, metric invariance, which indicates that the weight given to the different factor loadings is similar and, therefore, comparable between groups, was tested. However, only partial metric invariance was found for the cross-cultural comparisons - three items of the factor ‘Perceive and Understand Emotion’ did not hold for equal factor loadings across the different cultural samples. The fact that the three non-invariant items (Item 19, ‘When I meet an acquaintance, I immediately notice his/her mood’, Item 34, ‘I am able to tell the difference if my friend is sad or disappointed’, and Item 40, ‘I do not have difficulty noticing when somebody feels helpless’) were more related to the perceiving and understanding emotion in others could perhaps have a culturally based explanation. This result identifies the different interpretation of this aspect on both countries, thus revealing differences in the valence placed on perceiving and understanding others’ emotional aspects. The differences in these competencies could be due to the cultural appropriateness of the exhibited behavior by the two countries. Also, individuals’ subjective understanding and interpretation of the emotion-related experience could provide evidence for different cultural aspects.
The literature has supported the assertion that cultures featuring individualism-collectivism aspects tend to vary at recognizing and understanding emotions (Matsumoto, Reference Matsumoto1989). Perhaps Croatia, with its recent history of culturally based collectivism, has a more particular sense of group, places value in the emotional experience of others, and more frequently engages in behaviors that take into account the other’s perspective. Alternatively, Portugal may value a more self-centered and individualist perspective over the collective one, which could lead to a different interpretation or cultural appropriateness of these exhibited behaviors.
Moreover, a model of partial scalar measurement invariance was achieved in comparisons between Portugal and Croatia. Although the partial scalar model has improved the model fit (Δχ2 = 15.290, Δdf = 21, p = .083; ΔCFI = .001, ΔRMSEA = .006; cf. Table 7) and hold for this type of invariance, the fact that the scale has excluded several items in the three dimensions can compromise ESCQ’s validity assessment. The marginal non-significant p value and the minimal practical differences in CFI and RMSEA (compared to the unconstrained model) can perhaps indicate that the partial scalar model fit is being overestimated. Therefore, this particularly strong measure of invariance should be taken into account, since it might limit groups’ comparisons at this level. The invariant items at this level (Items 16 and 13 in PUE, Items 8 and 26 in ELE, and Items 30 and 39 in MRE) perhaps feature particular aspects of these emotional dimensions, which can be considered culturally and socially neutral since they have less influence on the interpretation of the intention or behavior.
The items that were excluded reflect the significant differences that were found to differentiate the way that both groups understood EI as a construct, particularly by the non-similarity on the use of the given scale. The absence of this strong measure of invariance may point towards the presence of some culturally driven response style (Mullen, Reference Mullen1995) and, consequently, a response scale shift on the two cultural groups. In fact, although the frequency-type scale that is used is accountable for less social vulnerabilities (Brown, Reference Brown2004) and the EI literature has evidenced difficulties in achieving such strong level of invariance (Ekerman et al., Reference Ekerman, Saklofske, Austin and Stough2011; Karim & Weisz, Reference Karim and Weisz2010; Li et al., Reference Li, Saklofske, Bowden, Yan and Fung2012; Parker et al., Reference Parker, Saklofske, Shaughnessy, Huang, Wood and Eastabrook2005), these dissimilarities may prove the differences in the way that both cultures acknowledge the frequency of their behavior or even the frequency of using these competencies on a daily basis, which consequently led to different interpretations of the scale reference values. Nonetheless, this particular case of invariance is difficult to prove but required to validate group-means comparisons.
Dissimilarities between both cultural settings may be accountable for the particular differences noted at the metric and scalar invariance levels on the present study. Given the probable influence of culture on the “Perceive and Understand Emotion” dimension highlighted by the results, further cross-cultural comparisons using this dimension should be conducted with caution. Future research involving the rephrasing of some items will likely improve the cross-cultural viability of this ESCQ factor. Moreover, the development of further studies with different measures (self-report vs. ability), featuring longitudinal designs, that examine the influence of both gender and different cultural settings on EI’s measurement invariance, particularly involving more countries and languages, should take place in the near future in order to add value to this promising EI field.
The current study had particular limitations. First, although every effort was made to endorse the comparability of the items, aspects related to the description and translation of the items’ content is obviously affected by the expression and experience of mood or feelings in every cultural group. Therefore, because of the nature of cross-cultural studies, the generalization of these results is limited to the analyzed contexts. Furthermore, the absence of proportionately distributed gender samples restrained further measurement invariance analysis, which would have contributed to the discussion of the gender invariance in these cultural groups. Moreover, the fact that the model was improved following different modification indices could have increased the probability of type I error (accept false positives). The use of iterative procedures of measurement equivalence (González-Romá, Hernández, & Gómez-Benito, Reference González-Romá, Hernández and Gómez-Benito2006), in this study, would have control this statistical vulnerability.
The findings from this research contributed to the clarification of current issues related to the cross-cultural validity of the ESCQ. In particular, in the field of EI, where the measurement invariance’s studies are still at an early stage, this study provided evidence of the partial metric invariance of ESCQ in the analyzed cultural contexts. Also, the results highlighted the effect of culture on aspects related to emotional experiences. In particular, the dissimilarities in some aspects of emotions’ perception and understanding for both countries raised awareness of the EI field to conveniently address the issues related to measures’ cross-cultural validity under penalty of misleading EI’s fundamental aim to study emotional experiences. Nonetheless, future cross-cultural research should be able to provide more comprehensive data for the validity of the ESCQ, as well as the scrutiny of the effect of culture on EI.
This research was supported by a Portuguese National Science Foundation Doctoral Grant (FRH/BD/72596/2010) awarded to Ana Costa and supervised by Luísa Faria.
