We are surrounded by a large amount of beautiful and provocative objects that we categorize as art. It has been suggested that these artistic manifestations provide a useful tool for neurosciences and should be able to yield valuable scientific insights (Zatorre, Reference Zatorre2005). How do these different objects of art influence each other? It seems obvious that art modulates human affective reactions and vice versa. Several psychological and psychophysiological experiments have repeatedly shown that pictures (e.g. Lang, Greenwald, Bradley & Hamm, Reference Lang, Greenwald, Bradley and Hamm1993) and music (e.g. Blood & Zatorre, Reference Blood and Zatorre2001; Krumhansl, Reference Krumhansl1997) can be used as stimuli capable of evoking emotional processes. Furthermore, cognitive appraisal seems to influence mood responses to music (Stratton & Zalanowski, Reference Stratton and Zalanowski1991) and music and painting appear to be effective for the induction of basic emotions (happiness, sadness, fear, anger and peace) in adults and can be judged according to pleasantness and arousal (Juslin & Sloboda, Reference Juslin and Sloboda2010; Kreutz, Ott, Teichmann, Osawa, & Vaitl, Reference Kreutz, Ott, Teichmann, Osawa and Vaitl2008). Music exists in all cultures and can be considered by most individuals as pleasurable stimuli acting as a potent reward (Salimpoor & Zatorre, Reference Salimpoor and Zatorre2013). Lundqvist, Carlsson, Hilmersson, and Juslin (Reference Lundqvist, Carlsson, Hilmersson and Juslin2009) compared the effects of happy or sad music over different components of the emotional response system and showed that music may induce genuine emotions through a process of emotional contagion. In addition, music elicited emotions that can modulate activity in limbic and paralimbic brain structures (Koelsch, Reference Koelsch2011). The judgment of beautiful or ugly paintings suggested a reciprocal interaction between motor and orbito-frontal cortex (Kawabata & Zeki, Reference Kawabata and Zeki2004) and affective pictures evoking fear or sadness activated the prefrontal cortex, mainly the right dorsolateral prefrontal cortical area (Baumgartner, Lutzb, Schmidt, & Jäncke Reference Baumgartner, Lutzb, Schmidt and Jäncke2006). Furthermore, a congruent emotional negative stimulus formed by painting and music affective stimuli presented simultaneously increased activity of structures involved in emotion processing (amygdala, hippocampus, parahippocampus, insula, striatum, medial ventral frontal cortex, cerebellum, fusiform gyrus; Baumgartner et al., Reference Baumgartner, Lutzb, Schmidt and Jäncke2006).
With the recording and storage of music, pictures and videos on electronic devices the simultaneous perception of music and painting has become a popular mode of enjoying the two types of artistic stimuli in daily life. Very little is known about how the responses to the two types of art interact and what are the emotional and cognitive consequences of their simultaneous presentation for information processing and memory. A large body of literature on crossmodal and heteromodal classical conditioning (e. g. see Razran, Reference Razran1958, Reference Razran1961 for a review; Dworkin, Reference Dworkin1993) with non-artistic material would suggest that the emotional valence of an unconditioned stimulus (US) following a conditioned stimulus (CS) is responsible for the emotional valence of the conditioned response (CR) independent of the stimulus modality. The tradition of classical conditioning in animals and humans rarely employed visual stimuli as US because of their inferior associative “power” to evoke a CR. There are exceptions from this rule if the visceral stimulus was previously paired with high intensity stimuli contexts or if the visual stimulus carries a particular evolutionary significance, as in the case of genetically prepared stimuli (Birbaumer & Öhman, Reference Birbaumer and Öhman1993; Öhman & Mineka, Reference Öhman and Mineka2001).
The study of reciprocal interactions between music and painting and their emotional interrelationships is the main aim of the present research. Although most scholars agree that viewers or listeners respond emotionally to musical (see Juslin & Laukka, Reference Juslin and Laukka2004) or pictorial stimuli, and these stimuli had been used in research to elicit affective states, the nature of the link between pieces of art and emotion remains a matter of some debate (e.g. see Konečni, Reference Konečni2008, Reference Konečni2013a). Konečni claims that the emotional impact of aesthetic objects have nothing to do with genuine emotions (such as anger, sadness, and joy) at most, these sublime artistic stimuli only induce minor physiological changes in viewers or listeners. The response induced by some visual artworks (called by Konečni aesthetic awe) is not a basic emotion but rather a mixture of basic emotions and these response form part of a tripartite hierarchy of aesthetic responses (Konečni, Reference Konečni2005). In Aesthetic Trinity Theory aesthetic responses could include being moved and aesthetic awe, often accompanied of physiological thrills, all three depending on the psychophysical, classical-conditioning and cognitive properties of aesthetic stimuli (e.g., Hunter, Schellenberg, & Schimmack, Reference Hunter, Schellenberg and Schimmack2010; Konečni, Reference Konečni2013b). Anyway in our experiment we are not studying directly basic emotions (such as fear or joy) but affective states recorded through scales measuring valence and arousal (Lang, Reference Lang1995; Lang, Bradley, & Cuthbert, Reference Lang, Bradley and Cuthbert1999). Pleasant responses reflect activation of the appetitive motivation system whereas unpleasant or disagreeable responses reflect activation of the avoidance motivation system (Thayer, Reference Thayer1967, Reference Thayer1989; Watson & Tellegen, Reference Watson and Tellegen1985). Sutton, Davidson, Donzella, Irwin, and Dottl, (Reference Sutton, Davidson, Donzella, Irwin and Dottl1997) pointed out the appetitive and aversive motivation system can be activated simultaneously and this creates a mix of positive (pleasant) and negative (unpleasant) affective states. Whether emotional reactions to art depend on cognitive processes (Stratton & Zalanowski, Reference Stratton and Zalanowski1991) and if those emotions are based on universal reaction patterns or are acquired during a process of individual acculturation is unclear (Grewe, Nagel, Altenmüller, & Kopiez, Reference Grewe, Nagel, Altenmüller and Kopiez2009). An interesting problem to solve is to know how the patterns of affective response interact to stimuli of different sensory modalities (e.g. pictorial vs musical). Parrott (Reference Parrott1982) studied the emotional effects of painting and music in three different conditions (music or paintings alone -producing strong affective responses or neutral ones- and music and painting combined) and he found interactions of emotional effects between painting and music. In Parrott's study emotional judgments included several dimensions (emotional/unemotional, happy/sad, beautiful/ugly, tense/relaxed, passive/active, liking/disliking, etc.). Affective states can also be studied using paintings or sounds differing in valence, i.e. (unpleasant to pleasant) and arousal (calm to high excitation; Lang, Reference Lang1995; Lang et al., Reference Lang, Bradley and Cuthbert1999). Concerning interactions between affective states Konorski proposed a mutually antagonistic relationship between pleasant and unpleasant stimuli when paired together. Following Konorski it can be argue that appetitive and aversive responses depend upon different, mutually antagonistic motivational systems interacting with each other. If a stimulus has pleasant appetitive value it will be difficult, but not impossible, to turn it into unpleasant and aversive stimuli and vice-versa. On the other hand, pairing stimuli with the same affective valence they should potentiate each other (Konorski, Reference Konorski1967, Mackintosh, Reference Mackintosh1974, p.23–24; Konorski & Szwejkowska, Reference Konorski and Szwejkowska1956). Whether these theoretical accounts explain interactions of valence and arousal of pictorial art and music remains to be investigated.
The hypothesis investigated here proposes we can modulate the affective value (valence and arousal) of paintings and music by pairing musical or pictorial stimuli of opposite affective values and getting their valence shifts to neutral. Moreover we can get some insight on emotional intensity effect of music over painting (or vice versa) in valence and arousal when the two stimuli are presented simultaneously.
EXPERIMENT 1: Affective Ratings to Musical and Pictorial Artistic Material
First we investigated affective experiences to a representative sample of musical and pictorial stimuli in order to establish an “affective metric” for the subsequent study of interactions. We wanted to explore the distribution of our musical and pictorial stimuli along the two dimensions of emotions according to the bio-informational model of valence (positive or negative emotions) and arousal (from calm to high excitability) proposed by Lang, Bradley, & Cuthbert (Reference Lang, Bradley and Cuthbert1997, Reference Lang, Bradley and Cuthbert1999). In Experiment 1 pictorial and musical stimuli were never presented simultaneously.
Method
Participants
Participants included 156 students (M age = 21,44 years, SD = 3,89 years, range = 16–35 years; 130 females and 26 males) that were recruited from the Conservatorio Profesional de Música de Alicante (16 males- average age 26, 21 females- average age 24.7) and the Facultad de Psicología de la Universidad Complutense de Madrid (10 males - average age 20.8 and 109 females- average age 20.2). Forty percent of the whole sample reported some musical education and only seven percent reported some pictorial artistic education.
Stimuli
The 52 musical fragments were selected according to various musical parameters such as tempo, volume, pitch, duration, rhythm, acceleration, articulation, harmony or instrumentation and the 52 paintings were selected having in mind pictorial attributes such color, form, perspective or iconographic and programmatic themes all of them (52+52) from a wide range of styles of East and West cultures and different epochs of art and music history. In particular timbre, pitch, amplitude and duration are the attributes that define how humans perceive musical sounds (Sadie & Grove, Reference Sadie and Grove1995) and these four attributes have proved to be useful to differentiate an isolated sound (Toharia et al., Reference Toharia, Morales, Juan, Fernaud, Rodríguez and DeFelipe2014) and in our musical excerpts rhythm, acceleration, articulation, harmony or instrumentation are also included. Having taken into account all these attributes has possibly allowed an adequate selection of different musical and pictorial stimuli because our stimuli were well distributed along all four possible quadrants of the two affective dimensions. In fact, the distribution of the valence and arousal ratings of painting and music stimuli is shown in Figure 1 and our results were similar to the distribution of the Affective Picture System described by Lang et al., (Reference Lang, Bradley and Cuthbert1997).
On the other hand, musical fragments contained a number of measures in order to have a significant musical meaning and the duration ranged from 3'' (stimulus number 34: 3'', Peter Heidrich, Happy Birthday to 38'' stimulus number 8: Ludwig Van Beethoven, Cavatina from String Quartet op. 130 (first bars) -average of 52 stimuli 15.6 sec-; e.g., Happy Birthday fragment selected included the following measures of 3 sec:
The paintings were presented for 3 sec. Among the selected paintings were Saturn devouring his children of Goya, two virgins of the Spanish Baroque painter José Antolínez, both almost identical but with different background (blue pale and navy blue); a detail of Vermeer’s The woman with red hat, The scream of Munch and three sketches of The sick child painted in different colors. We also used several paintings by Van Gogh (because of the versatility of his brush- stroke), Hokusai's Wave, a miniature picture of the Irish Book of Kells (VIII century); an image of a bull in the cave paintings of Lascaux Caves and contemporary paintings of Alejandro Franco. (See Appendix for a full list of paintings and musical excerpts).
Procedure
The task was self-administered. Participants were instructed to listen and to view the stimuli presented through online software hosted in the Campus Virtual UCM (Campos Bueno, de Juan, & Montoya Reference Campos Bueno, Juan and Montoya Jiménez2009). Participants were asked to rate arousal and valence components for each picture and musical fragment and the two types of stimuli were not shown simultaneously. Subjective responses were obtained through the Self Assessment Manikin (SAM) The Self Assessment Manikin is a non-verbal pictorial technique that measures affective response of individuals to a wide variety of stimuli (Lang, Reference Lang1995; Lang et al., Reference Lang, Bradley and Cuthbert1997, Reference Lang, Bradley and Cuthbert1999). The valence dimension ratings varied from completely happy (a smiling manikin rated as 9) to completely unhappy (a frowning manikin rated as 1). For arousal ratings, the scale varied from completely excited (a manikin with wide open eyes rated as 9) and completely calm (a manikin with closed eyes rated as 1).
The 52 musical fragments and the 52 paintings were counterbalanced and randomized. Each stimulus onset –painting or musical fragment- was programmed to start 3 sec after subject’s rating of previous painting or music stimulus. Although participants had no time restriction to rate the 104 stimuli, the experiment was finished after 45 minutes. Only data from participants who completed the experiment within these time limits were included in the statistical analyses.
Data analysis
To estimate the reliability of subjective ratings, mean valence and arousal scores were initially calculated for each stimulus across all participants and Cronbach´s alpha coefficients were computed separately for musical and pictorial stimuli.
For statistical analyses, mean valence and arousal ratings of music and paintings were also averaged across stimuli for each individual. In addition, participants were grouped according to their previous experience in art or not. Analyses of variance (ANOVAs) were used to assess the effects of TYPE of STIMULI (music vs. painting) and GROUP (experienced vs. non-experienced) on both arousal and valence ratings. For all ANOVAs, the Shapiro-Wilk and the Levene's tests were used for testing normality and homogeneity of variances, respectively. The statistical significant level was set at p < .05. All statistical analyses were carried out with SPSS version 19 (SPSS Inc., Chicago, IL, USA).
Results
The distribution of valence and arousal ratings for painting and music stimuli is displayed in Figure 1. Basically, musical and pictorial stimuli were located within all four possible quadrants of the two affective dimensions: high pleasant and high arousal, high pleasant and low arousal, low pleasant and high arousal, and low pleasant and low arousal.
Figure 1. Distribution of average ratings of musical and pictorial stimuli on valence and arousal. Participants rated 52 musical and 52 pictorial stimuli on valence and arousal. The average ratings of musical stimuli were more widely distributed over the two affective dimensions than ratings of pictorial stimuli. Music and pictorial stimuli marked in black were used in Experiment 2 as unpleasant stimuli, whereas stimuli marked in gray were used as pleasant stimuli. Stimuli marked in white were considered as neutral and were not used in Experiment 2.
In order to test whether the valence and arousal ratings for the 52 pictorial and 52 music stimuli were reliable, Cronbach´s alpha were calculated for each type of stimuli. In the case of valence ratings, the Cronbach's alpha coefficients were .86 for both painting and music stimuli. In the case of arousal ratings, the Cronbach's alpha coefficients were .88 for painting and .85 for music stimuli.
Table 1 display mean values of valence and arousal ratings for each type of stimuli (musical vs. pictorial) and each subgroup of participants according to their previous experience on art (experienced vs. non-experienced). The analyses of variance revealed that there were significant differences between musical and pictorial stimuli on both valence, F(1, 145) = 50.15, p < .001, and arousal ratings, F(1, 145) = 86.37, p < .001. No significant effects due to: (1) GROUP: valence: F(1, 145) = 1.35, p = .247; arousal: F(1, 145) = .19, p = .666; (2) the interaction between GROUP and TYPE of STIMULI, valence: F(1, 145) = .57, p = .451; arousal: F(1, 145) = 1.82, p = .179. were found.
Table 1. Ratings in valence and arousal of musical and pictorial stimuli (subgroups of subjects based on gender and previous experience on art or music)
EXPERIMENT 2: Interactions between Music and Painting
Experiment 1 revealed that music and painting stimuli could be allocated within the endpoints of the valence and arousal dimension of affective stimuli according with the model proposed by Lang et al. (Reference Lang, Bradley and Cuthbert1997). Furthermore, we observed that music and painting stimuli were reliable and that significant differences may exist between both types of stimuli regarding valence and arousal. The affective value obtained for each stimulus presented separately in Experiment 1, was used to study how music and painting interact when are paired. In Experiment 2, we analyze the reciprocal influence between our artistic stimuli by measuring changes in the affective value of these stimuli. Shifts in valence and arousal should be elicited by the association between musical and pictorial stimuli with the same or the opposite affective values (excluded the 20 stimuli classified as neutral ones). Because of the mutual influence of affective states induced by music and painting (as showed by e.g. Baumgartner, Esslen, & Jäncke, Reference Baumgartner, Esslen and Jäncke2006a) we expect that pairing of stimuli with similar valence would be rated as similar o even more pleasant (or unpleasant) than of one stimuli category alone. On the other hand, we also expect that paired stimuli with opposite valence should change the rating of pleasantness (or unpleasantness). The direction and intensity of these changes should be influenced by the affective power of music and painting.
Method
Participants
Participants included a new group of 202 healthy volunteer (M age = 21,35 years, SD = 3,57 years, range = 16–35 years; 166 females and 36 males) that were recruited from the Conservatorio Profesional de Música de Alicante (16 males -age average 26 years old- and 21 females -age average 25 years old-) and the Facultad de Psicología de la Universidad Complutense de Madrid (20 males with a mean age of 21 years old and 145 females with a mean age of 20.3 years old).
Stimuli
In Experiment 1 the art and music stimuli were presented separately but in the Experiment 2 stimuli were presented simultaneously in order to analyze the reciprocal influence between musical and pictorial stimuli with the same or the opposite affective values (that is, combining stimuli with similar values in valence and arousal; e.g. having high pleasant and high arousal, high pleasant and low arousal, low pleasant and high arousal, and low pleasant and low arousal).
The stimuli consisted of 32 videos created combining 32 pictorial and 32 musical fragments. No neutral stimuli were used in Experiment 2. Only stimuli with the highest valence values were selected to conform four groups of stimuli with different affective values: (pleasant music average 6.6 and pleasant painting average 5.9) and lowest valence values (unpleasant music average 4.0 and unpleasant painting average 3.7). Pleasant Groups A and B were formed having similar values of valence and arousal for music (Valence A1 = 6.6 and B1 = 6.7; Arousal A1 = 5.8 and B1 = 6.0) and for painting (Valence A2 = 5.8 and B2 = 5.9; Arousal A2 = 4.1 and B2 = 4.2). Unpleasant Groups C and D had also similar values of valence and arousal for music (Valence C1 = 3.9 and D1 = 4.1; Arousal C1 = 5.1 and D1 = 5.5) and for painting (Valence C2 = 3.6 and D2 = 3.8; Arousal C2 = 5.4 and D2 = 5.4).
In order to avoid a possible bias because of the different duration of our musical stimuli we took the option of simplify in only two lengths, 16 stimuli lasting 6 sec each and 16 long ones (14sec). The new musical excerpts - short or long- were obtained by duplicating and/or cutting the beginning or the end of the selected fragment but always keeping the significant musical meaning as referred in Experiment 1; e.g. now, in Experiment 2 the new Happy Birthday fragment was enlarged to reach 6 sec length:
Procedure
The task was self-administered. Participants were instructed to listen and to look at the videos presented through online software hosted in the Campus Virtual of the Universidad Complutense de Madrid. Participants were asked to rate separately valence and arousal elicited by the paintings or the musical fragments that appeared mixed together in 32 videoclips. Each videoclip lasted 6 sec or 14 sec (average 10 seconds) and music and painting were presented simultaneously. The duration of the music was 6 sec or 14 sec. The presentation of the painting always lasted three seconds. For each of the 32 clips the occurrence of the painting stimulus was counterbalanced, so that in half the cases, the painting appeared as the music begins (stimuli 1a, 2a, … 32a) and in the other half of the time the music and painting finalized simultaneously (stimulus 1b, 2b, … 32b). The 32 videoclips were distributed in four groups according to their valence (pleasant or unpleasant) and matched for arousal and duration:
Group A- Music and paintings of equal affective value –always pleasant- presented simultaneously subdivided as Group A1 (rating pleasant music –pM- that have been associated with pleasant painting –pP-) and Group A2 (rating the pleasant painting –pP- that have been associated with pleasant music –pM-);
Group B- Music and painting of opposite affective value –pleasant or unpleasant- presented simultaneously subdivided as Group B1 (rating pleasant music–pM- that have been associated with unpleasant painting –uP-) and Group B2 (rating the pleasant painting –pP- that have been associated with unpleasant music –uM-);
Group C- Music and painting of opposite affective value presented simultaneously subdivided as Group C1 (rating unpleasant music–uM- that have been associated with pleasant painting –pP-) and Group C2 (rating the unpleasant painting –uP- that have been associated with pleasant music –pM-);
Group D- Music and painting of equal affective value –always unpleasant- presented simultaneously subdivided as Group D1 (rating unpleasant music –uM- that have been associated with unpleasant painting –uP-) and Group D2 (rating the unpleasant painting –uP- that have been associated with unpleasant music –uM-);
Results
The effects of valence of music on painting or vice versa as a result of pairing were analyzed. The values obtained in this second experiment, where videos presented pairs of pictorial and musical stimuli, were compared with the ratings of the pictorial or musical stimuli of the first experiment presented alone. Table 2 summarizes the ratings of musical and pictorial stimuli when stimuli were presented isolated as music or as a painting (values obtained in Experiment 1), and when music and paintings were presented simultaneously in a videoclip (values obtained in Experiment 2).
Table 2. Sumary of values of musical and pictorial stimuli in valence and arousal
Notes: In Experiments 1 & 2 stimuli were always rated individually for Music value or Painting value –pleasant or unpleasant- and arousal after being presented isolated (Experiment 1) or when presented simultaneously forming a compound videoclip of Music + Painting (Experiment 2). (V = valence, A = arousal, t = time in seconds, M = Music, P = Painting, p = pleasant, u = unpleasant and see Figures 2&3).
The analysis of variance (ANOVA) on the change of valence on the musical stimuli when they were paired with pictorial stimuli in the videoclips shows the following results (see Figure 2A):
In Group A1, positive valence of pleasant music (V = 6.6) was reduced significantly (V = 6.1), that is, pleasant music became less pleasant when they were associated with pleasant paintings, F(1, 127) = 19.69, p < .001.
In Group B1, positive valence of pleasant music (V = 6.7) was reduced significantly (V = 5.8), that is, pleasant music became less pleasant when they were associated with unpleasant paintings, F(1, 127) = 5.24, p < .05.
In Group C1, negative valence of unpleasant music (V = 3.9) increased significantly (V = 4.2), that is, unpleasant music became less unpleasant when they were associated with pleasant paintings, F(1, 127) = 55.23, p < .001.
In Group D1, negative valence of unpleasant music (V = 4.1) did not change (V = 3.9), that is, unpleasant music remained with the same affective value when they were associated with unpleasant paintings, F(1, 127) = 0.96, p = .329.
Figure 2. Valence. A) Influence of painting on the valence of music. Subjective ratings of pleasant (M = 6.7) and unpleasant musical stimuli (M = 4.0) were clearly different when they were presented alone. The effects of painting on musical stimuli were different depending on the previous ratings of music. Thus, in the case of pleasant musical stimuli, the combination with pleasant (Group A1) and unpleasant (Group B1) paintings always elicited a shift towards neutral levels. By contrast, the ratings of unpleasant musical stimuli changed to neutral levels when paired with pleasant paintings (Group C1), but did not change when paired with unpleasant music (Group D1). B) Influence of music on the valence of painting. The valence of pictorial stimuli changed when they were paired with musical stimuli of the same or the opposite affective value. Subjective ratings of pleasant (M = 5.8) and unpleasant pictorial stimuli (M = 3.7) were clearly different when they were presented alone. The ratings of pleasant pictorial stimuli (Group B2) shifted to neutral levels when they were paired with unpleasant music. Moreover the ratings of unpleasant pictorial stimuli shift to neutral when paired with pleasant music (Group C2). By contrast, the ratings of unpleasant pictorial stimuli (Group D2) did not change when they were paired with unpleasant musical stimuli. Finally in the Group A2 the ratings of pleasant pictorial stimuli increased when they were paired with pleasant musical stimuli.
The analyses of variance (ANOVA) revealed that valence ratings of pictorial stimuli also changed after they were paired with music in the videoclips (see Figure 2B). In particular:
In Group A2, positive valence of pleasant paintings (V = 5.8) increased significantly (V = 6.0), that is, pleasant paintings became more pleasant when they were associated with pleasant music, F(1, 120) = 6.23, p < .05.
In Group B2, positive valence of pleasant paintings (V = 5.9) was reduced significantly (V = 4.8), that is, pleasant paintings became less pleasant when they were associated with unpleasant music, F(1, 120) = 68.63, p < .001.
In Group C2, negative valence of unpleasant paintings (V = 3.6) increased significantly (V = 4.9), that is, unpleasant paintings became less unpleasant when they were associated with pleasant music, F(1, 120) = 89.03, (p < .001).
In Group D2, negative valence of unpleasant paintings (V = 3.8) did not change (V = 3.8), that is, unpleasant paintings remained with the same affective value when they were associated with unpleasant music, F(1, 120) = 0.04, p = .850.
Finally we analyzed the influence of music on arousal of painting, or vice versa, as a result of the pairing stimuli with different affective values (pleasant or unpleasant valence) and matched in arousal. The analysis of variance (ANOVA) on changes of arousal of the musical stimuli when they were paired with pictorial stimuli revealed the following results (see Figure 3A):
Figure 3. Arousal. A) Influence of painting on the arousal of Music. In Group A1 the subjective ratings of arousal of unpleasant music goes down to calm and in Group B1 the arousal of pleasant music goes to calm as well when associated with unpleasant painting. There are no changes in arousal Groups C1 and D1 when paired with pleasant or unpleasant pictorial stimuli. B) Influence of music on the arousal of painting. In Group A2 the subjective ratings of arousal of unpleasant pictorial stimuli increases the excitation and in Group B2 the arousal of pleasant pictorial stimuli increases as well when associated with unpleasant music. There are no changes in arousal in Groups C2 and D2 when the pictorial stimuli are paired with pleasant or unpleasant musical stimuli.
In Group A1, arousal of pleasant music (A = 5.8) decreases significantly (A = 5.4), that is, music becomes more calmed when associated with pleasant paintings, F(1, 119) = 6.10, p < .05.
In Group B1, arousal (a) of pleasant music (A = 6.0) is reduced significantly (A = 5.6), that is, music becomes more calmed when associated with unpleasant paintings, F(1, 119) = 11.71, p < .001.
In Group C1, arousal (a) of unpleasant music (A = 5.1) does not change (A = 4.9) when associated with pleasant paintings, F(1, 119) = 1.18, p = .279.
In Group D1, arousal (a) of the unpleasant music (A = 5.5) does not change (A = 5.3) when associated with unpleasant paintings, F(1, 119) = 0.01, p = .934.
The analysis of variance (ANOVA) on changes of arousal of the pictorial stimuli when paired with musical stimuli shows (see Figure 3B) the following results:
In Group A2, arousal (a) of pleasant paintings (A = 4.1) increased significantly (A = 4.8), that is, paintings became more arousing when they were paired with pleasant music, F(1, 122) = 20.82, p < .001.
In Group B2, arousal of pleasant paintings (A = 4.2) increased significantly (A = 4.8), that is, paintings became more arousing when they were associated with unpleasant music, F(1, 122) = 19.82, p < .001.
In Group C2, arousal of unpleasant paintings (A = 5.4) did not change (A = 5.3) when they were paired with pleasant music, F(1, 122) = 0.01, p = .934.
In Group D2, arousal of unpleasant paintings (A = 5.4) did not change (A = 5.3) when they were paired with unpleasant music, F(1, 122) = 0.61, p = .610.
Discussion
Experiment 1 shows that our musical stimuli have a higher variability in arousal and valence ratings probably causing a stronger emotional effect of music over painting on dimensions of valence and arousal. Despite a selection of 52 paintings from almost all different periods and styles of history and our attempt to match dynamics, density and texture of the musical pieces with the paintings, the high arousal quadrants of painting, particularly the positive valence quadrant of paintings of the Wundtian dimension of arousal were underrepresented. Only the stimulus Saturn devouring his children, by Goya is rated as very unpleasant and has great arousal (V = 2.3 A = 7.5). Regarding the music only the stimulus The Murderer from Alfred Hitchcock's film Psycho (V = 3.0 A = 7.0) is near of the values reached by Goya's painting. But if we look at Table 1 it is clear that our musical stimuli are more distributed over the four quadrants than pictorial ones. This higher dispersion of our music stimuli may be due to that in Experiment 1, used to evaluate our stimuli, musical stimuli lasted longer than our paintings paints. On the other hand, music is a phenomenon that develops dynamically over time, while the pictorial observation is more static. This could constitute a selection problem or a consistent dominance of music over pictorial art. Anyway, despite the difficulties that may arise from our selection of stimuli, our data show that this does not preclude the study of the mutual interaction between music and painting because, when both stimuli are paired, the expected behavioral changes in the affective value of stimuli appear although the presence of painting in our video has a shorter duration than the music. Replications with different selections of stimuli will clarify the problem of representativeness. The strong effect reported and the large sample used may however indicate a natural emotional “superiority” of music.
Certainly our results are limited to a sample of students with similar age range and studying similar courses (Experiment 1: 156 participants, M age = 21,44 years, SD = 3,89 years, range = 16–35 years; 130 females and 26 males; Experiment 2: 202 participants, M age = 21,35 years, SD = 3,57 years, range = 16–35 years; 166 females and 36 males) but our data seems to be in agreement with other studies with less with less male participants than females and a wide range of age (e.g. Grewe et al., Reference Grewe, Nagel, Altenmüller and Kopiez2009; 38 participants, M age = 37.9 years, SD = 15.6 years, range = 11–72 years; 29 females and 9 male). Students of music and psychology did not differ in their ratings. Musician’s preference for music therefore cannot explain this effect. Our data yielding no differences on valence and arousal ratings due to gender or previous experience with pictorial art or music seems to be in agreement with previous studies showing that level of musical training (Rickard, Reference Rickard2004) or gender (Grewe et al., Reference Grewe, Nagel, Altenmüller and Kopiez2009; Parrott, Reference Parrott1982) had no influence in subjective ratings or in physiological reactions to musical stimuli. Baumgartner et al. (Reference Baumgartner, Lutzb, Schmidt and Jäncke2006) studied evoked emotions of happiness, sadness or fear by presenting simultaneously congruent emotional music and pictures showing that music had a strong emotional enhancement effect of the emotion evoked by affective pictures. Indeed, our Experiment 2 provides an indirect confirmation of the stronger affective “power” of music. Unpleasant as well as pleasant music carries the valence of paintings towards the valence of the musical piece. A classical heteromodal conditioning effect preferring auditory US stimuli over visual seems unlikely because pairings of music and paintings were counterbalanced in valence and arousal over trials, music preceding the paintings as often as paintings’ presentation started with the music. Another counter argument may involve the longer duration of presentation of the musical pieces (6 or 14 sec versus 3 sec). A musical piece however needs more time to become a perceptual Gestalt than a painting which involves simultaneous binding of contrasts and features through high frequency gamma oscillations of the brain (Lutzenberger, Preissl, Birbaumer, & Pulvermüller, Reference Lutzenberger, Preissl, Birbaumer and Pulvermüller1997) while musical features develop through successive associative binding processes in the auditory system with different oscillatory frequency characteristics (Kaiser, Lutzenberger, Preissl, Ackermann, & Birbaumer, Reference Kaiser, Lutzenberger, Preissl, Ackermann and Birbaumer2000). Exact matching of the presentation interval may therefore ignore the basic neuronal nature of feature binding and Gestalt formation of the two modalities (Singer & Gray Reference Singer and Gray1995). Still, replication of these experiments with different and identical presentation time of the stimuli is necessary.
Another argument against a stimulus duration effect is the strong negative, probably distracting effect of painting-valence on music valence: pleasant music becomes neutral (valence ratings shifts to 5) if paired with pleasant paintings; the iso-directional effect of valence of music on painting is reversed here in a strong neutralizing effect of pleasant painting. In addition, pleasant music is also neutralized by negative valence paintings. Thus, painting may touch more on the arousal dimension than the valence of music.
These results are somewhat different than the effects of emotional visual material particularly the IAPS (International Affective Picture System, Lang et al., Reference Lang, Bradley and Cuthbert1999) on acoustic startle amplitude: negative valence potentiates startle, while positive valence attenuates it (Patrick, Bradley, & Lang Reference Patrick, Bradley and Lang1993). The photographic nature of the IAPS may create more valence variation than our artistic paintings in most people and thus affect auditory perception (startle) more readily. A comparison of our experimental procedure with IAPS slides may thus produce different effects of music and visual material on each other. Pictures of art derive their emotional meaning at least in part from educational and history of art knowledge sources while music has an immediate emotional effect less filtered through education and experience particularly in non-musicians. Non-musicians and lay persons concerning pictorial art may respond to the IAPS photographs more strongly than to works of art as used in our experiment. Comparing the Wundt-quadrants of Fig. 1 for our pictorial material and the IAPS-slides in an unselected sample of healthy persons (see Lang Reference Lang1995) clearly shows a much stronger representation of the upper right quadrant of highly arousing positive slides for the IAPS than the artistic material mostly due to sex pictures. Still, the under-representation of highly arousing positive paintings in our material cannot explain the neutralization of pleasant music by both pleasant and unpleasant paintings as well the positive effect of pleasant music on unpleasant paintings whose representation in the high arousing negative quadrant is comparable to the IAPS slides’ distribution.
If we focus on the mutual interaction of valence of music and painting the results of behavioural changes in valence can be summarized as follows. The first main conclusion is regarding the influence of pictorial stimuli on valence of music. In that sense, our study shows that: (1) pleasant music changes to neutral (valence value shifts to 5) when is associated with pleasant painting (Group A1); (2) moreover, pleasant music changes to neutral when is associated with unpleasant painting (Group B1); (3) furthermore unpleasant music changes to neutral when is associated with pleasant painting (Group C1); and (4) finally unpleasant music does not change when is associated with unpleasant painting (Group D1). The second main conclusion is regarding the influence of musical stimuli on the valence of painting. In that sense, our study demonstrates that: (1) pleasant painting changes to more pleasant when is associated with pleasant music (Group A2); (2) moreover, pleasant painting changes to neutral when is associated with unpleasant music (Group B2); (3) the unpleasant painting changes to neutral when associated with pleasant music (Group C2); and (4) unpleasant painting does not change when associated with unpleasant music (Group D2).
The results may also be in part explained by conditioning through pairing two stimuli, supposedly neutral, but with emotional acquired values such it has occurred through the cultural experiences in our pieces of music and paintings. We can observe that, when musical or pictorial stimuli of opposite affective values are paired then their valence shifts to neutral. It can be seen that valence of pleasant stimuli (Groups B1 and B2) decreases its affective value to neutral when paired with negative stimuli. And valence of unpleasant stimuli (Groups C1 and C2) increases its affective value to neutral when paired with positive stimuli. However, when paired stimuli are both unpleasant the affective value of stimuli remains in its original negative valence (Groups D1 and D2). If we look at the interaction between music and painting when both stimuli are pleasant we found that when pleasant music is associated with pleasant painting it can be seen that affective value (Group A1) then affective value of music don’t increases its affective value, on the contrary valence of music shifts to neutral. However when pleasant painting is paired with pleasant music (Group A2) then affective rating value of pleasantness of pictorial stimuli is enhanced and affective value of painting increases showing the power of music on painting valence. It is, when pleasant music is associated with pleasant painting the ratings on painting does not move to neutral, but on the contrary the effect of music produces an increasing of the affective value of pictorial stimuli. Painting stimuli are less arousing than musical fragments and it could explain changes in arousal when pleasant painting or pleasant music interacts with musical or pictorial stimuli. We found that arousal of pleasant music always decreases when pleasant music is paired with pleasant (Group A1) or unpleasant painting (Group B1) but arousal of unpleasant music it is not affected by pleasant (Group C1) or unpleasant painting (Group D1). But when pleasant painting is paired with pleasant (Group A2) or unpleasant music (Group B2) arousal of painting always increases. Finally arousal of unpleasant painting it is not affected by pleasant (Group C2) or unpleasant music (Group D2). It means that agreeable painting increases its arousability when interacts with positive or negative music but when agreeable music interacts with positive or negative painting its arousability decreases.
To summarize: despite the described critical aspects of representativeness of the selected material for all types of artistic creation and the unresolved problem of duration of stimulus presentation the data reported here provide preliminary evidence in a large sample of a stronger and overriding effects of valence carried by music than pictorial art.
