The sport environment is a context of competence and achievement where motivational factors generated by significant others play an important role in the effects of sport participation on children and youths’ psychosocial development (Smith, Smoll, & Cumming, Reference Smith, Smoll and Cumming2009). During childhood, parents’ influence seem to be a crucial factor in the athletes’ sport experience. However, during adolescence, influences from coaches and peers, especially on young people’s perceptions of competence, become more significant, while the parents’ impact decreases (Boixadós, Valiente, Mimbrero, Torregrosa, & Cruz, Reference Boixadós, Valiente, Mimbrero, Torregrosa and Cruz1998; Chan, Lonsdale, & Fung, Reference Chan, Lonsdale and Fung2012). In this line, previous literature has taken interest in analyzing the influence of the team environment created by coaches and team-mates on youth athletes (e.g., Keegan, Spray, Harwood, & Lavallee, Reference Keegan, Spray, Harwood and Lavallee2010; Reinboth & Duda, Reference Reinboth and Duda2006). Two major motivational theories, Achievement Goal Theory (AGT; e.g., Duda & Hall, Reference Duda, Hall, Singer, Hausenblas and Janelle2001) and Self-Determination Theory (SDT; e.g., Deci & Ryan, Reference Deci and Ryan2000), have focused on this issue.
Within the study of team environment, AGT is aimed particularly at motivational climate (e.g., Vazou, Ntoumanis, & Duda, Reference Vazou, Ntoumanis and Duda2006) and SDT at autonomy support (e.g., Adie, Duda, & Ntoumanis, Reference Adie, Duda and Ntoumanis2008). Motivational climate refers to a person’s perceptions of the environment motivational indicators and expectancies (Ames, Reference Ames, Meece and Schunk1992). According to this author, motivational climates are defined in terms of mastery and performance. Later studies used the terms task-involving and ego-involving to describe mastery and performance climates respectively (e.g., Newton, Duda, & Yin, Reference Newton, Duda and Yin2000). In a mastery (or task-involving) climate, success is defined as individual effort and improvement. In contrast, in a performance (or ego-involving) climate, the focus of learning is on interpersonal comparison and evaluation is based on normative standards (for a review, see Ntoumanis & Biddle, Reference Ntoumanis and Biddle1999). Vazou et al. (Reference Vazou, Ntoumanis and Duda2006) studied the relation between ego and task climates promoted by both coaches and peers. Their results showed that ego and task coach-created climates were positively related to ego and task peer-created climates.
Within the AGT, some studies have selected goal orientation as a personal variable in which team environment has an influence (e.g., Balaguer, Castillo, Duda, & García-Merita, Reference Balaguer, Castillo, Duda and García-Merita2011). Goal orientation is defined as the predominant dispositional goal in achievement situations, depending on how people evaluate their success and interpret their ability (Nicholls, Reference Nicholls1989). Two major goal dispositions have been proposed: task orientation and ego orientation. When task-oriented, individuals perceive their ability as self-referenced and focus on personal improvement, task mastery and exerted effort. When ego-oriented, perceptions of ability are other-referenced and individuals need to show superiority to feel competent (for a review, see Duda & Ntoumanis, Reference Duda and Ntoumanis2003). Nicholls (Reference Nicholls1989) viewed task and ego orientations as orthogonal and later studies confirmed this aspect (e.g., Balaguer, Castillo, & Tomás, Reference Balaguer, Castillo and Tomás1996). Moreover, repeated exposure to certain climates can lead to subsequent modifications of task and ego orientations. In fact, previous research has shown how athletes’ task and ego orientation were respectively influenced by coaches’ (e.g., Smith et al., Reference Smith, Smoll and Cumming2009) and peers’ (e.g., Vazou, Reference Vazou2010) task-involving and ego-involving climates. Also, earlier studies related these constructs to the regulations defined in the SDT (e.g., Deci & Ryan, Reference Deci and Ryan2000) and found that task orientation predicted intrinsic motivation and identified regulation, and ego orientation predicted introjected and external regulations (e.g., Ntoumanis, Reference Ntoumanis2001).
In the framework of SDT, autonomy support defines how the others allow and encourage initiative and freedom of decision, and share the players’ vision in solving problems (Gagné, Ryan, & Bargmann, Reference Gagné, Ryan and Bargmann2003). Usually, research has focused on coaches’ autonomy support and has found that it positively predicts athletes’ self-determined motivation (e.g., Álvarez, Balaguer, Castillo, & Duda, Reference Álvarez, Balaguer, Castillo and Duda2009). However, to the best of the authors’ knowledge previous works have not studied peers’ autonomy support (for an exception, see Ramis, Torregrosa, Viladrich, & Cruz, Reference Ramis, Torregrosa, Viladrich and Cruz2013). In Ramis et al. study (2013), peers’ autonomy support only had a slight effect on self-determined motivation. Moreover, results showed that coaches’ and athletes’ autonomy support were moderately correlated.
In this context, a number of questionnaires have been developed to assess the influence of team environment on athletes’ performance, affect or cognition. However, studies have increasingly needed more items as theories and analyses have become more complex (see results of our review below). This requirement lengthens the time participants spend on completing the questionnaires, and it can lead to fatigue, boredom and apathy, especially in children and youths. For this reason, the time used filling in the instruments might affect the quality of data obtained (e.g., the number of missing values). This issue has appeared not only in the study of the team environment but also in the whole arena of sport psychology research, as we will refer to later.
Given this scenario, it is clear that there is a need to improve the methodology for obtaining quantitative data in sport psychology research. To this end, short-forms of questionnaires appear to be useful in this field, where shortening has not been a common practice (for an exception, see Terry, Lane, Lane, & Keohane, Reference Terry, Lane, Lane and Keohane1999). In addition, it is worthwhile noting that shortening an existing questionnaire has some advantages over creating a new shorter instrument (Coste, Guillemin, Pouchot, & Fermanian, Reference Coste, Guillemin, Pouchot and Fermanian1997) because: (a) it enables phases of instrument development (e.g., item pool composition) to be bypassed; and (b) the instrument will appear familiar to users of the original form.
Although not common in sport psychology, within the field of clinical and health psychology shortening has been a widespread practice (e.g., Cox et al., Reference Cox, Oliver, Rial-González, Tomás, Griffiths and Thompson2006; Mühlan, Bullinger, Power, & Schmidt, Reference Mülhan, Bullinger, Power and Schmidt2008) and has helped to obtain a more economical and efficient diagnosis. The economy of diagnosis refers to the length reduced and time saved with the short-form compared with its original. Thus, short-forms make the response process more amenable, decreasing the burden on participants and lessening the resources spent on the study. From our point of view, research in sport psychology should learn from this perspective. An analysis of the quantitative papers about Sport Psychology published in 2010 in the four Sport Psychology journals with the highest impact factor (i.e., Journal of Sport & Exercise Psychology, Psychology of Sport and Exercise, The Sport Psychologist and Journal of Applied Sport Psychology) and in the four Spanish journals with the highest impact factor (i.e., The Spanish Journal of Psychology, Psicothema, Anales de Psicología and Revista de Psicología del Deporte) showed that researchers usually ask young athletes to respond to big amounts of items (e.g., athletes under 18 years old answered an average of 64 items in the international journals; detailed information about this review is available from the first author).
Efficiency balances economic effort with the loss of information or validity (Mülhan et al., Reference Mülhan, Bullinger, Power and Schmidt2008). When using instruments assessing health-related quality of life, efficiency is particularly important to ease clinical trials and clinical practice (Moran, Guyatt, & Norman, Reference Moran, Guyatt and Norman2001). However, shortening per se could entail losses in questionnaires’ psychometric properties, especially in terms of reliability and content validity (for a review on the sins of short-forms development, see Smith, McCarthy, & Anderson, Reference Smith, McCarthy and Anderson2000). Thus, the shortening process should include a careful selection of items in order to minimize these potential losses. As recommended by Coste et al. (Reference Coste, Guillemin, Pouchot and Fermanian1997), this selection should mainly consider content-driven criteria, and data-driven criteria should only be regarded when theory arguments are not enough to make a choice. Watson and Clark (Reference Watson and Clark1997) addressed the issue of how much a questionnaire might be shortened by and reported that less than four items per factor would yield an insufficient internal reliability (see also rationale by Jokovic, Locker, & Guyatt, Reference Jokovic, Locker and Guyatt2006). However, our review of the papers published in 2010 in the Sport Psychology journals and in the Spanish research journals with highest impact factor revealed that participants had to respond to many more items than just four per factor.
Developing economical and efficient questionnaires (i.e., short-forms) is important not only because it reduces the resources participants and researchers spend in the study, but also because shortening questionnaires might help to improve the quality of quantitative data, in terms of: (a) decreasing the number of missing values; (b) diminish the number of aberrant response patterns (i.e., “persons with item score patterns that are unexpected”; Meijer & Sijtsma, Reference Meijer and Sijtsma1995, p. 262), and (c) improving response rate (Edwards, Roberts, Sandercock, & Frost, Reference Edwards, Roberts, Sandercock and Frost2004).
Considering the benefits that shortening can bring to sport psychology research and the fact that no previous works in sport psychology have aimed to study how short-forms could help to improve data quality, our purpose was to develop short-forms to assess motivational climate, autonomy support and goal orientation. We wanted these short-forms to retain the core of the main dimensions of the AGT constructs (i.e., task and ego). According to Vazou et al. (Reference Vazou, Ntoumanis and Duda2006), focusing on the main points of the task (e.g., effort) and ego (e.g., social comparison) dimensions, instead of assessing the specific aspects of each climate, would enable the comparison between goal orientation and motivational climates, which could not be done with the original instruments (see also Whitehead, Andrée, & Lee, Reference Whitehead, Andrée and Lee2004).
In summary, the main goal of this work was to develop short-forms of four instruments assessing motivational factors from the AGT (i.e., peer-created motivational climate and goal orientation) as well as the SDT (i.e., coach and peer autonomy support) and provide evidence of their psychometric properties. Specifically, our study focused on the Peer Motivational Climate in Youth Sport Questionnaire (PeerMCYSQ; Ntoumanis & Vazou, Reference Ntoumanis and Vazou2005), the Sport Climate Questionnaire (SCQ; Deci, Reference Deci2001) and the Task and Ego Orientation in Sport Questionnaire (TEOSQ; Duda, Reference Duda1989). We also wanted to find out if these short-forms could improve data quality (i.e., less missing values and aberrant response patterns), as hypothesized.
We carried out three different studies. Study 1 describes the development of the short-forms and includes the selection of items (version 1 of the short-forms), judgmental validation and adjustment (i.e., expert meetings and focus groups; version 2). Study 2 focuses on the validation of these instruments and their iterative improvement (versions 3 and 4). Study 3 validates version 4 in a new sample and also includes structural equation modeling (SEM) in order to assess criterion validity. Figure 1 describes the entire process.
Figure 1. Flowchart describing the process followed in the studies 1, 2, and 3.
Study 1: Short-form development
The main purpose of Study 1 was to develop the short-forms of the PeerMCYSQ and the TEOSQ. To do so, we used both qualitative (i.e., expert meetings and focus groups) and quantitative (i.e., sample of young athletes) data sources. Also, we wanted to test the existing short-form of the SCQCoach and to see if the same structure could be applied to the SCQPeer.
Method
Participants
We obtained qualitative and quantitative data from participants in Study 1 to develop the short-forms. The qualitative stage involved an expert committee and two focus groups. On one side, the expert committee included one female methodologist, two male applied sport psychologists, one female and five male researchers in sport psychology, and two male and one female youth sport coaches. On the other, 17 male athletes (age range: 12–16 years old) participated in the focus groups.
Quantitative data were obtained from 114 youth athletes (M age = 14.65, DT age = 2.10, age range: 10–19) from the Barcelona area. This sample included more male athletes (61%). All of them played team sports, either in local or regional competitions. We will refer to these participants as Sample 1.
Instruments
We administered the original versions of the questionnaires with the aim of obtaining the quantitative data that would help us to develop the short-forms.
Peer motivational climate
To assess the peer-created motivational climate, the Sample 1 responded to the Spanish version (Moreno et al., Reference Moreno, Conte, Martínez, Alonso, González-Cutre and Cervelló2011) of the Peer Motivational Climate in Youth Sport Questionnaire (PeerMCYSQ; Ntoumanis & Vazou, Reference Ntoumanis and Vazou2005). Twelve items belonged to the task factor and nine to the ego factor. The original study provided evidence for their reliability and internal structure (Ntoumanis & Vazou, Reference Ntoumanis and Vazou2005), confirmed in a Spanish sample (Torregrosa et al., Reference Torregrosa, Viladrich, Ramis, Azócar, Latinjak and Cruz2011). An average of 8 minutes was needed to respond to the PeerMCYSQ.
Autonomy support
We assessed coach autonomy support using the 15-item Sport Climate Questionnaire (SCQ; Deci, Reference Deci2001) in its Spanish version (Balaguer, Castillo, Duda, & Tomás, Reference Balaguer, Castillo, Duda and Tomás2009). Reliability evidence was provided in previous studies (e.g., Balaguer et al., Reference Balaguer, Castillo, Duda and Tomás2009). To assess peer autonomy support, we administered the 15-item SCQ adaptation for peers (SCQ Peers; Ramis et al., Reference Ramis, Torregrosa, Viladrich and Cruz2013). Ramis et al. (Reference Ramis, Torregrosa, Viladrich and Cruz2013) provided evidence for its reliability. The participants (Sample 1) spent an average of 5 minutes to complete each SCQ.
Achievement goal orientations
To assess the participants’ disposition to task and ego, they responded to the Task and Ego Orientation in Sport Questionnaire (TEOSQ; Duda, Reference Duda1989), which has been adapted into Spanish by Balaguer et al. (Reference Balaguer, Castillo and Tomás1996). Seven items belonged to the task factor and six to the ego factor. Previous studies provided evidence for the TEOSQ psychometric properties in different cultures (e.g., Li, Harmer, Chi, & Vongjaturapat, Reference Li, Harmer, Chi and Vongjaturapat1996), and in the Spanish population (e.g., Balaguer et al.). An average of 5 minutes was necessary to fill in this questionnaire.
Procedure
The development of the short-forms followed five steps (see upper part of Figure 1): (1) we generated theory-driven criteria to obtain content validity; (2) we administered the original questionnaires and we analyzed the data to get data-driven criteria; (3) we selected the items to generate the short-forms due to the theory-driven and data-driven criteria (version 1); (4) we conducted focus groups to know participants’ views of the questionnaires, which is a source of validity evidence based on the response process (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999); and (5) experts analyzed focus groups results and consensus was reached about the items wording (version 2).
Our shortening process of the PeerMCYSQ and the TEOSQ was initially based on the methodological tips proposed by Coste et al. (Reference Coste, Guillemin, Pouchot and Fermanian1997). On the selection of the items that would be included in the short-forms, we firstly considered theory-driven criteria, and we used data-driven criteria to refine this selection. Our theory-driven criteria came from a literature review and the advice of a group of experts. Literature review included highly-cited papers in this field (e.g., Duda & Nicholls, Reference Duda and Nicholls1992) and the number of times an item had been previously used as example item in relevant papers. The development of the theory-driven criteria was led by the first author, advised by the heterogeneous group of experts. Two important decisions were made: (1) to create short-forms with four items per factor; and (2) to choose the items belonging to the core of the dimensions task and ego, with the aim of allowing comparisons between goal orientation and motivational climates. The group of experts considered task dimension in terms of (a) skills improvement, (b) mastery and (c) effort, and ego dimension as (a) outplaying the others, (b) showing better skills and (c) comparison to the others. Consequently, although original PeerMCYSQ includes five lower order factors, short-form was not expected to retain all the breadth of the construct. Based on the theory-driven criteria, experts selected between five and six items that were considered most relevant to define each construct (i.e., peer-created motivational climate and goal orientation).
Simultaneously, we collected data from Sample 1. At least two researchers attended each administration session. The participants were told that participation in the study was voluntary, were assured about confidentiality and were instructed to respond as honestly as possible (for more details, albeit in Spanish language, see Ramis, Torregrosa, Viladrich, & Cruz, Reference Ramis, Torregrosa, Viladrich and Cruz2010). Data were collected during the last two months of the season, to ensure that goal orientation and perceptions of the motivational climates and autonomy support had been established.
Once data were collected, data-driven criteria were obtained from results of internal reliability and internal structure. More information about these analyses can be found in results section. The data-driven criteria helped us determine which combination of items worked better (version 1 of the short-forms). Version 1 contained eight items for both the PeerMCYSQ and the TEOSQ, including four items assessing each ego and task factors. SCQPeer adopted the structure of the short SCQCoach and included six items assessing only one dimension.
The shortening process continued with two focus groups that were conducted in order to find out the opinion of our target population. First, focus group participants completed Version 1 of all the short-forms and marked sentences or words that they did not understand. Then, we ensured that participants knew the meaning of each item and we helped them to propose changes in the items wording to make them easier to understand. After the focus groups, we met with the experts and they discussed the changes the young players had proposed until consensus was reached (Version 2). Version 2 had only little differences compared to Version 1 (i.e., refinements in the items wording; more information is available from the first author). These changes (e.g., synonyms) did not modify the original meaning of the items.
Results
We assessed reliability (i.e., Cronbachs’ alpha coefficients, alpha without the item and correlation between items) and internal structure (i.e., results from exploratory and confirmatory factorial analyses) in Sample 1. For the sake of brevity, details of these results are available from the corresponding author. However, Cronbachs’ Alpha values are presented in Table 2.
Discussion
In Study 1 we developed the short-forms by using theory-driven as well as data-driven criteria. Short-forms for the PeerMCYSQ, the SCQPeer and the TEOSQ were created. We also worked with the SCQCoach short-form. A future study was needed to validate these short-forms.
Study 2: Preliminary psychometric validation of the short-forms
The main purpose of Study 2 was to validate the short-forms. Thus, we wanted to assess their internal structure, reliability and concurrent validity. As a consequence of this assessment, we found that short-forms needed some improvement, and then we conducted expert meetings and focus groups to do so.
Method
Participants
In Study 2 we collected quantitative and qualitative data. Sample 2 comprised 309 young athletes (M age = 14.19, DT age = 1.73, age range: 11–19) from the Barcelona area and was used to validate the short-forms. The higher proportion of boys (66%) was consistent with the sport context where our work was placed (García Ferrando & Llopis, 2006). As we wanted to study peer autonomy support and motivational climate, all selected participants regularly played team sports (35% basketball, 35% futsal, 16% volleyball and 14% handball). Participants took part either in local or regional competitions. Qualitative data were obtained from the same group of experts as in Study 1 and from two focus groups with seven female (age range: 10–12) and eight male (age range: 12–13) athletes.
Instruments
Our participants responded to Version 2 of our short-forms. The PeerMCYSQ included eight items, containing four items assessing task motivational climate and four assessing ego motivational climate. Both the SCQCoach and the SCQPeer included six items. The TEOSQ comprised eight items, with four of them assessing task orientation and four ego orientation.
All the instruments were answered on a 7-point Likert scale (1 = strongly disagree / completamente falso; 7 = strongly agree/completamente verdadero), although the TEOSQ originally had a 5-point scale. We changed its response range according to the tips proposed by Streiner and Norman (Reference Streiner and Norman2008) for the construction of continuous scales. By giving the scales the same range we wanted to improve the quality of the results by easing the participants’ answering process. Additionally, as Spanish language has minor changes when referring to a female or a male (e.g., adjective suffix), a version for each gender was administered.
In order to assess concurrent validity of the short-forms, participants from the Sample 2 also responded to questionnaires concerning coach-created motivational climate (Newton et al. Reference Newton, Duda and Yin2000; adapted to Spanish by Balaguer, Guivernau, Duda, & Crespo, Reference Balaguer, Guivernau, Duda and Crespo1997), perceived competence (retrieved from McAuley, Duncan, & Tammen, Reference McAuley, Duncan and Tammen1989; adapted to Spanish by Balaguer, Castillo, & Duda, Reference Balaguer, Castillo and Duda2008) and autonomy (adapted to Spanish from Standage, Duda, & Ntoumanis, Reference Standage, Duda and Ntoumanis2005), as well as intrinsic motivation and introjected regulation (Lonsdale, Hodge, & Rose, Reference Lonsdale, Hodge and Rose2008; adapted to Spanish by Viladrich, Torregrosa, & Cruz, Reference Viladrich, Torregrosa and Cruz2011). Further information about these instruments and their psychometric properties in our sample is available from the corresponding author upon request.
Procedure
First, the Sample 2 completed the Version 2 of the short-forms. Data collection followed the same protocol as in Study 1. Confidentiality was guaranteed throughout. After the data analyses (see middle part of Figure 1), experts were consulted in order to clarify the issues that did not work well in the analyses and they proposed changes to improve the short-forms (Version 3). Then, we conducted two focus groups to ask the target population about the changes we made and to look at the items wording. The experts met again and accepted the changes (Version 4).
Results
Data quality comparison
The data quality was assessed in terms of the number of missing values and aberrant response patterns (e.g., alignment errors). Table 1 shows a comparison between the quality of the data obtained with the original versions of the PeerMCYSQ, the SCQPeer, the SCQCoach, and the TEOSQ, and with their short-forms (Sample 2). Although athletes in Study 1 had responded to the complete versions, those 114 participants were not enough to do this comparison. Consequently, data were obtained from previous studies of our research group that had used the same administration protocol with samples that were considered equivalent to those of this work. A total of 648 cases had responded to the complete questionnaires. Analyses showed that short-forms had less missing values, with some of the constructs having all values completed. Results of a t test revealed that using the short-forms we obtained significantly less missing responses per item: t(655) = −4.01, p < .001, d = −0.13, 95% CI [−0.19, –0.01]. The analyses of the aberrant response patterns did not show relevant results and consequently are not presented. Moreover, Table 1 shows the time needed to respond to the original instruments and to the short-forms. On average, participants completed the short-forms in a third of the time spent on the complete versions.
Table 1. Response Time and Data Quality Comparison (Study 2)
Note:
Average time indicates how many minutes a participant usually needs to complete each questionnaire. It was calculated as the completion time needed by 95% of participants averaged across the administration sessions.
Descriptive statistics and reliability
Table 2 presents the descriptive statistics and Cronbach’s Alpha coefficients for the scales in Study 2 (Sample 2). Also, we offer reliability coefficients from the Study 1 sample (α study1 ) to facilitate comparison. The mean value of the variables displays a desirable pattern of a positive sports experience: high peers’ task climate, moderate-to-high coach and peer autonomy support, and high task goal orientation. All of these means were well above the midpoint of the scale range. The scales did not present any relevant skewness or kurtosis, indicating relative normality. The only exception was task goal orientation, which showed slight negative skewness and small positive kurtosis. All the internal reliability coefficients from Sample 2 ranged from .70 to .85, thus achieving Nunnally’s (Reference Nunnally1978) above .70 criterion for psychological scales. Additionally, nearly all the items contributed to the Alpha coefficient for their dimensions. Only deletion of Item 4 from the PeerMCYSQ (i.e., “Criticize their team-mates when they make mistakes”) made the Alpha coefficient increase.
Table 2. Descriptives (Sample 2) and Internal Consistency for each Measure
Note:
Range for all variables is 1-7. MC = motivational climate. α study2 = α of data from Study 2; α study1 = α of data from Study 1.
For the imputation of missing values, we replaced them with the participant’s mean on the factor that included each particular missing value. According to Graham’s criterion (2009), this approach should not have consequences on the data analyses because missing values were less than 5% of all data points.
Confirmatory Factor Analyses (CFA)
Confirmatory Factor Analyses (CFA) were conducted with MPlus 5.2 to assess the adequacy of the data to the a priori models, using the Weighted Least Squares Means and Variance Adjusted (WLSMV), which utilizes a diagonal weight matrix with robust standard errors and a χ²-test adjusted by mean and variance (Muthén & Muthén, Reference Muthén and Muthén2008). We present the following fit indices: the χ2, the Root Mean Square Error of Approximation (RMSEA), the Comparative Fit Index (CFI), and the Tucker-Lewis Index (TLI). According to Jackson, Gillaspy and Purc-Stephenson’s criterion (2009), a good fit to the model might be considered when CFI and TLI values are close to .90 and the RMSEA value is close to .08. In the best scenario, the χ 2 test is also expected not to be statistically significant. CFA using data from Sample 2 were conducted. The results in Sample 2 were not satisfactory and leaded us to work in the improvement of the short-forms. The two-factor model did not fit to the PeerMCYSQ data well: χ2(19) = 146.18, p < .001, CFI = .87, TLI = .88, RMSEA = .20. The CFA for the SCQCoach also revealed inadequate factor structure: χ2(9) = 88.17, p < .001, CFI = .94, TLI = .89, RMSEA = .17. The SCQPeer was found to have an acceptable factor structure: χ2(9) = 35.43, p < .001, CFI = .98, TLI = .97, RMSEA = .08. The two-factor model showed a correct fit to the TEOSQ data, except for RMSEA: χ2(19) = 74.28, p < .001, CFI = .93, TLI = .94, RMSEA = .13. All the factor loadings were statistically significant and above .40. Modification indices revealed two issues: (1) Item 4 from PeerMCYSQ (i.e. “Criticize their team-mates when they make mistakes”), which was predicted to be an item of the ego factor, was inversely related to the task factor (i.e., had a negative factor loading on task dimension); and (2) the TEOSQ items from the ego factor grouped in pairs, differentiating those referring to myself from those referring to the peers. Exploratory Factor Analyses (EFA) results led to the same conclusions. These results are not reported here but are available from the corresponding author. Due to these results and the qualitative data obtained from expert meetings and focus groups, we chose a different item from the original instrument and we included it in Version 4 of the PeerMCYSQ (i.e., “Complaint when the team doesn’t win”) in replacement of Item 4.
Correlations between variables
First, analysis of the bivariate correlations between short-forms dimensions showed that: (a) Task and ego peer-created motivational climates were moderately correlated (r = .33, p < .01) and (b) task and ego goal orientations were almost not correlated, although the coefficient was significant (r = .11, p < .05), supporting the TEOSQ orthogonal structure.
Table 3 contains the bivariate correlations between factors of the short-forms and variables that were hypothesized to be related to. All the expected correlations were positive, statistically significant (p < .01) and ranged from .19 to .58, with most of the values being above .30. It may be considered that all these correlations were low (.20 < r < .40) or moderate (.40 < r < .60). Specifically, task goal orientation had a low correlation to perceived coach’s and peers’ task-involving climates and to perceived competence, and had a moderate correlation with intrinsic motivation. Ego goal orientation, in turn, had a low correlation with perceived coach-created ego climate, introjected regulation and competence, and a moderate correlation with peer-created ego-involving climate. Also, low correlations appeared between coach’s task-involving climate and his/her autonomy support, and between peers’ task-involving climate and their autonomy support. Finally, perceived autonomy correlated slightly to coach and peer autonomy support.
Table 3. Bivariate Correlations between Study 2 Variables
Note:
Correlations with p > .01 are not shown. Correlations showing evidence of concurrent validity of the short-forms are presented in bold. GO = goal orientation; MC = motivational climate. Line separates between short-forms and other measures.
**p < .01.
Discussion
In Study 2 we began the process of validating the short-forms developed in Study 1. Evidence concerning their reliability, concurrent validity and improvement of data quality was obtained. However, the results of their internal structure were not entirely satisfactory. Thus, experts meetings and focus groups were used with the aim of improving the short-forms. A future study was then necessary to test the last version of the short-forms (Version 4).
Study 3: Psychometric validation and criterion validity
The main purpose of Study 3 was to test the psychometric properties of Version 4 in a new sample. Moreover, we wanted to assess criterion validity using Structural Equation Modeling (SEM).
Method
Participants
Sample 3 comprised 204 participants (M age = 12.48, DT age = 1.72., age range: 9–18). One-hundred and ten of them were female athletes (54%). Athletes’ competed in football (53 %) or synchronized swimming (47 %).
Instruments
Participants responded to Version 4 of our short-forms, including the PeerMCYSQ, the SCQPeer, the SCQCoach, and the TEOSQ. We also assessed motivational regulations using the Spanish Version (Viladrich et al., Reference Viladrich, Torregrosa and Cruz2011) of the Behavioral Regulations in Sport Questionnaire (BRSQ; Lonsdale et al., Reference Lonsdale, Hodge and Rose2008).
Procedure
Data from Sample 3 were obtained with the aim of validating Version 4 of the short-forms (see lower part of Figure 1). The data collection procedure remained the same as those outlined in the previous studies.
Results
Confirmatory Factor Analyses (CFA)
Table 4 presents the fit indices for all the short-forms. All of them showed a better fit to the models compared to the previous CFA in Study 2 and were found to have an acceptable factor structure. Only the CFA for the PeerMCYSQ did not meet the RMSEA criteria proposed by Jackson et al. (Reference Jackson, Gillaspy and Purc-Stephenson2009). All the items had statistically significant factor loadings above .50 and can be found in Table 5.
Table 4. Fit Indices for the CFA (Sample 3)
Note:
df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CFI = Robust Comparative Fit Index; TLI = Tucker-Lewis Index.
**p < .01.
Table 5. Items and Factor Loadings that Comprise the Short-forms (Sample 3, Version 4)
Note:
Contains Spanish and English wording of the items included in Version 4 of the short-forms. The Spanish wording is not a direct translation, as both experts and focus groups participants proposed changes to make the items more understandable.
Criterion validity
Analyses of Structural Equation Modeling (SEM) were performed in order to explore the criterion validity of our short-forms. Specifically, we examined two different models, one for the Achievement Goal Theory and other for the Self-Determination Theory. The first one studied the effect of the peer-created motivational climate on goal orientation, which in turn had an effect on motivational regulations (Figure 2). According to the previous literature, we expected task and ego peer-created motivational climates to correlate and to have an influence on both task and ego orientations. The task orientation was hypothesized to positively predict self-determined types of motivation (i.e., intrinsic motivation, integrated regulation and identified regulation). The ego orientation was expected to have a positive effect on controlled motivation (i.e., introjected and external regulations) and amotivation. The second model examined the influence of peers’ and coaches’ autonomy support on motivational regulations (Figure 3). In this model, we expected perceptions of peers’ and coaches’ autonomy support to correlate. We hypothesized that the positive effect of coaches’ autonomy support on self-determined types of motivation and negative effect on controlled regulations and amotivation would be bigger than peers’. In both models self-determined types of motivation were expected to correlate highly. Also, controlled regulations and amotivation were hypothesized to have high correlations.
Both models displayed a good fit to the data: AGT model, χ2(708) = 1003.01, p < .001, CFI = .93, TLI = .93, RMSEA = .05; SDT model, χ2(566) = 756.20, p < .001, CFI = .95, TLI = .95, RMSEA = .04. As can be seen in the Figures 2 and 3, the models showed most of the hypothesized paths. However, there were some exceptions. In the AGT model, peer-created task climate moderately predicted adopting an ego orientation. Also, ego orientation positively predicted identified regulation, a type of self-determined motivation. In the SDT model, perceptions of coach’ autonomy support did not have a negative effect on controlled regulations or amotivation. Moreover, peers’ autonomy support did not predict self-determined types of motivation.
Figure 2. Structural Equation Model Based on the Achievement Goal Theory
Note. For presentation simplicity purposes only significant paths and correlations are showed, and item indicators are not presented. Discontinuous lines show negative paths. *p < .05; **p < .01; ***p < .001.
Figure 3. Structural Equation Model Based on the Self-Determination Theory
Note. For presentation simplicity purposes only significant paths and correlations are showed, and item indicators are not presented. Discontinuous lines show negative paths. *p < .05; **p < .01; ***p < .001.
Discussion
Study 3 had two purposes: On the one side, results from the CFA in Study 3 demonstrated that the internal structure of the short-forms improved due to the changes made in Study 2. On the other, results from structural equation modeling analyses provided evidence concerning the short-forms criterion validity.
General Discussion
We have developed short-forms of the PeerMCYSQ, the SCQPeer and the TEOSQ and we also have tested the SCQCoach. Some areas of research such as Health and Clinical Psychology have been making use of the benefits of shortening questionnaires for a number of years. However, within Sport Psychology few researches have focused on this issue. Regarding this point, we discuss the evidence supporting the merit of these short-forms, the limitations of our work and propose future lines of research.
Our short-forms have some advantages over the original instruments because: (a) they place less burden on athletes and researchers, (b) are economical (i.e., length reduced and time saved), (c) are efficient (i.e., little loss of information or validity compared to the resources saved) and (d) help to improve the data quality. Our results showed that our participants only needed a third of the time used to respond to the original questionnaires to fill in the short-forms. Also, the number of missing values decreased significantly when using the short-forms.
Psychometric merit of the short-forms
Shortening per se could induce losses in the psychometric properties of questionnaires, especially in terms of reliability and content validity (Coste et al., Reference Coste, Guillemin, Pouchot and Fermanian1997; Smith et al., Reference Smith, McCarthy and Anderson2000). Consequently, Coste et al. (1997) recommended that content validity and reliability should particularly be assessed. In this work we also provided validity evidence based on the response process and we assessed internal structure and external validity. Our item selection included theory-driven criteria obtained from literature review and experts’ advice, which are sources of validity evidence based on test content (American Educational Research Association et al., 1999). Following Patton’s (Reference Patton2002) recommendation that in order to show diversity, as many opinions as possible have to be considered, the experts meetings included applied sport psychologists, researchers in Sport Psychology, methodologists and youth sport coaches. This heterogeneity also enhanced decisions about the representation and relevance of items. As stated by the American Educational Research Association et al. (1999), participants’ views of the questionnaires are a source of validity evidence based on the response process. Thus, we conducted two focus groups so as to reassure that participants’ responses stick on the meaning of the constructs. That is, the wording of the items was revised to ensure that the participants understood their meaning. Also, adapting items’ wording probably helped to reduce the time participants spent on responding to the short-forms, because completion time is influenced by items’ comprehensibility (Terry et al., Reference Terry, Lane, Lane and Keohane1999).
The analyses of reliability and internal structure also presented evidence supporting the merit of our short-forms. With regard to reliability, all the short-forms reached Nunnally’s criterion (1978) with Cronbach’s alpha values being above .70. These results are quite interesting because a lower internal consistency could have been expected due to the shortening. Moreover, Streiner and Norman (Reference Streiner and Norman2008) argued that constructs’ heterogeneity could diminish their own internal consistency. However, all our short-forms presented a satisfactory internal consistency.
Referring to the internal structure, the CFA presented an acceptable fit to the hypothesized models. Initially, the results from data in Study 2 did not reach the criteria proposed by Jackson et al. (Reference Jackson, Gillaspy and Purc-Stephenson2009). Then, we made changes to the short-forms (Version 4). The CFA conducted with data from Study 3 confirmed that the short-forms have an acceptable internal structure. It must be said that although the PeerMCYSQ did not show a poor fit to the model, it only reached two of three criteria proposed by Jackson et al. As said by Ntoumanis and Vazou (Reference Ntoumanis and Vazou2005) “validation is an ongoing process” (p. 19) and consequently we recommend future studies to work on and improve the short-form of the PeerMCYSQ. We propose the same process that we previously followed in Study 2 to improve the short-forms (i.e., expert meetings and focus groups).
Results also provided evidence supporting short-forms criterion validity (i.e., analyses of correlations and structural models). In Study 2, evidence supporting the concurrent validity of the instruments came from the low to moderate bivariate correlations between short-forms and variables that they were hypothesized to be related to. As we expected, we did not find high correlations because those other variables were constructs in the nomological network (i.e., the interlocking system of laws that constitute a theory; see Cronbach & Meehl, Reference Cronbach and Meehl1955) not conceptually equivalent to the short-forms constructs. In Study 3, we performed two different structural models. In the model from the AGT, task and ego peer-created motivational climates predicted athletes’ goal orientation, which is congruent with previous studies (e.g., Vazou, Reference Vazou2010). This goal orientation, in turn, had an effect on motivational regulations. In line with Barkoukis, Ntoumanis, and Nikitaras (Reference Barkoukis, Ntoumanis and Nikitaras2007), task orientation positively predicted experiencing self-determined motivation and had a negative effect on external regulation and amotivation. Moreover, ego orientation positively predicted identified regulation and both types of controlled motivation (i.e., introjected and external regulations). In our study, ego orientation also had a small effect on amotivation. Viewed globally, these results support the “adaptive role of high task goal orientation in promoting self-determination in sport” (Ntoumanis, Reference Ntoumanis2001, p. 407). In the model from the SDT, results showed that coaches’ autonomy support positively predicted athletes’ self-determined motivation, as hypothesized (e.g., Pelletier, Fortier, Vallerand, & Brière, Reference Pelletier, Fortier, Vallerand and Brière2001). However, paths of peers’ autonomy support were not totally congruent with previous literature (Ramis, Torregrosa, Viladrich, & Cruz, Reference Ramis, Torregrosa, Viladrich and Cruz2013).
Inevitably, short-forms cannot include the whole content of a construct. Thus, our short-forms cannot assess the specific aspects of peer-created motivational climate, autonomy-support or goal orientation. Consequently, they should be used carefully. We propose that our short-form would be useful for those who want a quick measure (i.e., applied practitioners) or those who work with a model with multiple variables, and consequently need efficient measures. If the purpose is to study the peer-created motivational climate, the autonomy-support or the goal orientation in detail, researchers should work with the complete versions.
We recommend future studies to focus on two issues. Firstly, it would be interesting to use the short-forms to construct other models with multiples variables. As structural equation modeling results did not totally support SCQPeer criterion validity, researchers could compare results from short and complete versions. They could generate a model including other dependent variables expected to be related to it (e.g., psychological needs satisfaction) and compare results with the ones obtained with the complete version. Secondly, researches could continue testing how these short-forms can improve data quality. As completing short-forms requires less time, a lower number of missing values and aberrant response patterns and a higher response rate are expected (Edwards et al., Reference Edwards, Roberts, Sandercock and Frost2004). In this work, results revealed that the short-forms had significantly less missing values. However, no aberrant response patterns were found. Independent confirmation about the improved quality of data could be obtained if all users of these short-forms took a simple habit: To report percentage of observed missing values, possible aberrant response patterns and response rates in their samples.
Conclusion
We have shown the benefits of using the short-forms of the PeerMCYSQ, the SCQPeer, the SCQCoach and the TEOSQ and we have provided evidence supporting their psychometrical merit. We have also confirmed that these short-forms can help to improve the quality of the data, as less missing values were obtained. Therefore, these short-forms might facilitate better understanding of the team environment, both in research and applied practice, although further research is needed to demonstrate their efficiency in studying the team environment and its relations with other variables. In short, these short-forms show: (a) similar psychometric properties to those from the original questionnaires can be obtained, while placing less of a burden on athletes and psychologists; (b) ease in data management; and (c) improvement in data quality by registering less missing values. We hope our results will encourage others in the Sport Psychology field to focus on the shortening practice, in order to improve the quality of research and applied practice.
