Virtual reality is one of the buzzwords of late twentieth- and early-twenty-first-century popular culture. Video games, online environments and virtual theme parks offer users the experience of a different world outside ordinary life. Music plays a significant role in both the establishment of and immersion in such virtual worlds. In order to investigate precisely how the sonic dimension of virtual reality works, this chapter describes and theorises various forms of musical virtual reality, arguing that the virtual worlds of music challenge existing understandings of virtual reality and immersion. Musical virtual reality is an omnipresent, perpetually moving, physical transmission of what will be conceptualised here as ‘musical energy’.
Once upon a Dream of J. S. Bach
Virtuality is a non-manifest dimension of reality that escapes, precedes or exceeds actualisation: the virtual is separated from manifest reality by its not-being-actualized (Lévy Reference Lévy1997, 91–3). Virtual reality refers to a simulated timespace which is experienced as a reality but which is different from the day-to-day world. While it has become a key concept in academic and societal debates since the emergence of digital media (with ‘cyberspace’ as the most often-discussed form of virtual reality), Marie-Laure Ryan argued in her influential book Narrative as Virtual Reality (Reference Ryan2001) that virtual reality comes into being by immersion in any form of narrative, analogue or digital: the immersion in the simulated world of a film or a book can be just as deep as that in a video game or online environment. The media generating virtual reality can be textual, visual, auditory or olfactory – any physical sense or any combination of physical senses can be involved in the perception of a reality that is different from, but parallel to, the day-to-day world that we call ‘reality’. Reading the Grimm brothers’ fairytale ‘Sleeping Beauty’, we can believe ourselves to be wandering around in its world of magic spells and mysteriously growing forests; seeing a film adaptation of the story, we see this world alive before our very eyes, and are drawn into it by enchanting visuals; playing a Sleeping Beauty video game we interact with the inhabitants of this world; and visiting the Dutch theme park De Efteling we can see the princess asleep in her castle, which we approach over a rocky forest path. All of these versions of a fairytale world are as real as our senses tell us they are, and all of them can be as immersive as our perceptual attention allows. Each is a form of virtual, not actual, reality, simulated, but tangible, enveloping, captivating.
Sound and music are key contributors to any form of virtual reality. Because of their remarkable affective and mnemonic power, even the smallest sounds can evoke worlds of connotations, emotions and memories. These associative capacities are employed in very precise ways in any form of simulated reality. Tchaikovsky’s music for the Sleeping Beauty ballet was chosen for Disney’s 1959 animated movie: it was such an integral part of the film’s world that the chosen tagline was ‘Wondrous to see – Glorious to hear’, and the film’s song ‘Once upon a dream’ has become a musical icon for cartoon magic. All Disney films and video games are celebrated for their immersive soundtracks, and the company prides itself in its use of musical world-building. Disney has brought out two Sleeping Beauty video games, Princess Aurora’s Singing Practice and Enchanted Melody, in which the gameplay itself revolves around music as the player ‘helps’ Princess Aurora practise her singing or dance through the magic forest from the film.
For other audiences, Sleeping Beauty is not necessarily connected with Tchaikovsky, but with J. S. Bach. The Fairytale Forest in the Efteling theme park is dotted with large red and white toadstools that play the Minuet in G major from Anna Magdalena Bach’s notebook (BWV 114). The piece’s harpsichord timbre, triple metre and soothingly simple progressions blend perfectly with the surroundings of this ‘magic forest’. I remember leaning my ears on these toadstools as a little girl and being transported into a world of fairytale wonder – aided, no doubt, by the fact that I had been listening to a Grimm tales LP at home in which ‘Sleeping Beauty’ was accompanied by Bach’s fourth Brandenburg Concerto (BWV 1049). In these examples, listening to Tchaikovsky’s or Bach’s music is partaking of a fairytale virtual reality. The examples also show that such musical virtual realities are simultaneously immersive and subjective, coloured by personal listening histories and musical connotations.
If musical virtual reality is not limited to digital contexts, if it is created by such subjective agencies as personal connotations, and if, in fact, any musical experience can potentially engender a kind of virtual experience, then how, if at all, can it be described objectively?
Schizophonia, Mobile Music, Video Games
Schizophonia
The invention of sound recording in the late nineteenth century separated sound from its physical origin, enabling sound to be replayed at another time and in another space than those of its first occurrence. The mere fact of this separation revolutionised sonic communication by creating disembodied sound worlds that were perceived by many at the time as disconcerting. As recording technology definitively unmoored sound from its origins in time and space, acousmatics became a reality: it was now possible to hear a sound without perceiving where it came from, so that voices appeared to be ghosts, music appeared to be spectral, and the dog Nipper famously took his master’s recorded voice for his actual presence (Sterne Reference Sterne2003, 287–333). The spectral aspects of sound recording were explored in early horror film, which exploited audiences’ unease with the fact that the moving image was now accompanied by disembodied voices (Van Elferen Reference Van Elferen2012, 34–47). In terms of virtual reality, early sound recording enhanced the cinematic experience by layering the visual world with an auditory counterpart: film sound completed the illusion that cinema goers were steeped in another world, that the cinema really did take them out of their own reality and into another, temporary but utterly bewitching, world of audio-visual stimulation. That that world was evidently disembodied but simultaneously sensually perceivable, and therefore convincingly real, only enhanced the immersion in the uncanny realm of horror in which spectral sounds evoke the return of repressed anxiety (van Elferen Reference Van Elferen2012, 24–33, 173–90).
Reducing the occult overtones of such earlier assessments and applications, R. Murray Schafer coined the term ‘schizophonia’ for the spatial and temporal separations established by sound recording (Reference Schafer1977, 88). He argues that with the advent of ever more sophisticated playback equipment, such as 3D surround sound, ‘any sound environment [can] be simulated in time and space … Any sonic environment can now become any other sonic environment’ (91). Sonic virtual reality, therefore, is not limited to the context of film or television soundtracks, in which music is relatively subservient to visual content, but can exist in and of itself at any given time. Whenever we turn on the radio, press ‘play’ on a stereo set, or switch on our mobile music device, we bring into being a schizophonically simulated reality, a virtual reality that consists of sound alone (cf. Voegelin Reference Voegelin2014, 62–84, 153–6).
Like any other virtual reality, these sonic worlds exist parallel to the everyday world: driving to work while listening to Lucia di Lammermoor on my car CD player, I am as immersed in Donizetti’s rendering of Lucia’s torment as I am in traffic. Because immersion in sonic virtual reality does not hinder other sensory perceptions, moreover, musical realities easily mingle with everyday reality. Tia DeNora’s (Reference DeNora2000) research on music’s personal and social functions illustrates this blending of worlds. Her interviewees explain that when the daily routines of driving to work, cooking dinner or taking a bath are accompanied by music, the affective connotations of that music overlay the experience of such prosaic activities, lending the allure of other possible worlds to their own world. And because music’s ‘stickiness’ makes it retain its connotations whatever the context (Abbate Reference Abbate2004, 523), the mixture of ‘real’ and ‘virtual’ goes even further: having listened to Donizetti on a number of my commutes, I now find that in fact the motorway to work itself reminds me of Lucia, as if the opera were the soundtrack to my journey and the two types of perception have in fact blended into one reality that is half-real and half-virtual (cf. Bull Reference Bull2007, 87–93).
Mobile Music
The example of car radio illustrates how the blending of realities through music becomes especially noticeable in mobile music culture. The invention of the Sony Walkman in 1979 and the Apple iPod in 2001 caused a seismic shift in the sociology of music consumption. If listening to music in the car turns the motorway or cityspace into a film projected onto your windscreen, walking through cities and landscapes with the same music in your earbuds resembles actually participating in that film (Bull Reference Bull2007, 38–49). As Schafer predicted in 1977, sound recording has ultimately led to ‘the portability of acoustic space’ (Reference Schafer1977, 91). But this acoustic space never consists of sound alone: it is always, and inevitably, filled by the connotations evoked by that sound. The timespace of any music consists of sound and is inhabited by the listener’s memories, emotions and identifications; the agency of these unconscious processes galvanises acoustic space, renders it alive and real. For this reason musically simulated realities never just exist parallel to, but are always inextricable parts of, everyday life.
Moreover, the fact that mobile devices are carried on the body changes the physical dimension of the listening experience. We listen to our music device as we walk through town, cycle to the station, or exercise in the gym: the beats and rhythms of the music synchronise or syncopate with the movements of our bodies, and often we adjust either our playlist or the speed of our movements in order to create a better match between them. Thus, the choreography of bodies moving in time to music is no longer reserved to dance alone, but also characteristic of such activities as walking, cycling and exercising. Because of mobile music, listening culture has become a newly embodied culture (cf. DeNora Reference DeNora2000, 75–108). As a result, immersion in sonic virtual worlds is not merely emotional or mnemonic, but physical and geographical as well. Mobile music marks the consolidation, the personalisation and the vicarious re-embodiment of schizophonia: the separation of sound from its origin is no longer uncanny, but a natural fact of life in the age of the iPod; it enables every listener to move around in their own private capsule of sonic virtuality; and as this virtuality is narrowly aligned with the listener’s own movements, the disembodiment of schizophonia is replaced by the listener’s post facto, vicarious re-embodiment of recorded sound.
From time to time concerns are raised about the alleged anti-social aspects of mobile music culture. If everyone moves through life in their own auditory bubble, the argument goes, the very possibility of social interaction is thwarted. However odd it may seem to see a train full of people with their headphones on and staring at their mobile phone, the truth of the matter is, of course, that all these private bubbles are connected in myriad ways to other private bubbles. As Imar de Vries’s archaeology of the mobile phone has shown, mobile communication is driven by an idealistic discourse that promises a ‘communication sublime’: a ‘final and universally accessible communication space’ in which anyone can communicate with everyone else at any place and at any time (Reference De Vries2012, 17–18). The enormous success of social media – Facebook had 2.23 billion active users in the second quarter of 2018 – attests that global connectivity simply changes rather than diminishes the way in which we experience communication. Facebook, Twitter, Instagram and all the other social networks have added a new layer of reality that sits between real life and virtual reality: the constantly accessible social reality of connectivity.
Music distribution and consumption have rapidly moved into the connected space of social media. Listeners recommend the music they like to each other on various social media platforms and also link up their actual listening experiences through such streaming services as Spotify, Shazam and SoundCloud. These online radio stations enable users to listen to the same music as our friends do, and to let their musical tastes be guided by those of listeners whom the programmes’ algorithms calculate to be like-minded people. Here the virtual worlds of mobile music listening, with their intriguing blend of associative, physical and geographical characteristics, move from the private to the public domain. Musical experience is no less personal in the shared zone of social listening than it was in the private space of individual playlists: on the contrary, precisely because streaming software collects data on our musical preferences it enables us to share our most personal emotions and memories. Listening to my indie Spotify playlist, I am immersed in my own musical preferences; I can share them with others; by following Spotify’s recommendations I may get to know new people who like similar music; I may get to know music I did not know hitherto; and I am able to share these new insights with my friends, old and new, so that we can all immerse ourselves in the same virtual world of musical experience. All of this can be done while I sit on the train with my headphones on and stare at my mobile phone, and while my private bubble extends to any place in the world. Musical social networks accumulate all the aspects of musical virtual reality discussed so far, and add the social element of sharing and bonding. As such, the connected space of streaming services represents a mediated counterpart of the concert hall: not in real life but also not outside it, this simulated reality is as real as our own emotions and as physical as our own bobbing head, and it reaffirms the strong social ties of shared musical interests in online communities.
Video Games
Although the musical virtual reality of sound recording, film soundtracks, mobile devices and social media is immersive, it is not as interactive as that in video games. Gaming interaction gives players the sense that they are actively participating in the game’s virtual world. Whether they operate the arrow keys of their PC in order to move two-dimensional blocks in Tetris or the wireless Dualshock controller of their PS4 in order to navigate avatar Arno Dorian through Assassin’s Creed Unity’s meticulously designed game space, they appear to be in control of the action. Compared to reading, viewing or listening, video gaming leads to different kinds of immersion and virtual reality because of its interactive nature. Involving not only more physical senses than books or films but also active participation and playful intervention in a non-linear story, immersion and virtuality in video games would seem to be more all-encompassing than those in other media. It is not surprising, therefore, that the academic literature on both topics has grown exponentially since interactive video games started to dominate the entertainment industry.
Next to appealing graphic design and engaging gameplay, sound and music play a crucial part in gaming virtual reality. Game sound design has several functions at once, some very similar to those of film and television sound. Sounds can help build a convincing game space: just as in film or on television, diegetic sounds such as ‘natural’ bird song or ‘industrial’ white noise help the player determine the kind of environment through which she navigates, and non-diegetic sounds such as ‘sad’ minor melodies or ‘dangerous’ pounding drums can help her identify the mood that the environment is supposed to evoke. Equally comparable to film and television sound, non-diegetic game sound gives the player clues regarding the characters or situations they meet in this virtual world. Evil characters are introduced by dissonant harmonies, battle scenes are underscored by loud and fast music, and victories are identified by heroic orchestral cadences. Such themes often work in a similar way to leitmotifs in that they are instantly recognisable – so recognisable that they verge on clichés – sonic identity markers like the monstrous grunts and metallic drones of the Necromorphs in Dead Space 2. So-called ‘interface sound’ indicates when players have completed a level or picked up ammunition, and guides them through the game’s menus: this kind of sound immerses the player in the virtual matrix of gaming software itself (Collins Reference Collins2008, 127–33). Finally, like film and television music, game music is often used as a sonic branding device, the auditory identity kit of a game in which all its essential elements are wrapped. A famous sonic game brand is the 2002 theme of the Halo series, which combines Gregorian chant and rising string motifs with syncopated drum rhythms and electric guitar riffs. The theme evokes the nostalgia of historical adventure as well as the thrilling suspense of first-person shooter gameplay: it is precisely because of this unusual combination of musical connotations in the even more unusual context of a science fiction game that this theme music has become a celebrated sonic game brand.
But just as the virtual reality of video games differs from that in other media, so too does its sonic counterpart.1 Unlike film and television narratives, games do not follow linear story-lines. Instead, the development of gameplay depends on the speed, direction and success with which a player moves through the game. The ensuing non-linear, interactive and individual development of gameplay cannot be underscored by linear music but is accompanied by what is called ‘adaptive audio’, in which themes and motifs loop in non-linear patterns so as to match the non-linear gameplay progress (cf. Collins Reference Collins2008, 125–30, 139–65). In the case of Halo, composer and producer Martin O’Donnell and Michael Salvatori isolated the various components of the theme – the chant, the violins, the guitar – so that they could be used as loops accompanying different parts of the gameplay. This non-linear form of composing has important consequences for the gaming experience as well as for the discussion of musical immersion and virtual reality. As adaptive audio is tied to particular gameplay situations, game sound and music are not just heard but activated by the player. It is her navigation that brings these sound worlds into being: entering certain spaces cues monster leitmotifs to begin, backing off to load her gun makes the music fall silent again, her victory over the monsters fades the battle music into victory music. As many games are designed with 3D sound, players are able to locate precisely where in the virtual space monsters are lurking, and can let their avatar act accordingly: game music has become a Gaming Positioning System for the navigation of gaming virtual reality (van Elferen Reference van Elferen2011, 32–4). The music does not merely accompany the playing experience, it is the playing experience: the epic victory themes, for instance, that have made the Final Fantasy soundtracks so famous represent the end of the player’s personal journey, battles and triumph.
Game musical connotations and interaction converge in the sonic establishment of gaming virtual reality. While the associative power of Halo’s plainchant voices and the syncopated drum rhythms are powerful contributions to the game’s virtual reality, the interactivity of this same music gives the player active agency in the construction and development of that reality. Sonic immersion in Halo simultaneously entails, on the one hand, immersion in your own world of musical connotations and its intermedial connections with the graphic game design, and, on the other, immersion in the flow of gameplay, which allows you to open up layer upon layer of music. The speed of the gaming flow necessitates relatively unambiguous musical associations: it would be very confusing if Dead Space 2’s Necromorphs were not accompanied by grunts and electronic drones but by mellifluous harp and piano motifs – the player would be too late in her assessment of this sonic information, lose the battle, and the game would end. Because in-game musical interpretation must be fast, game music often seems somewhat clichéd in its signifying patterns. Fantasy role-playing games (RPGs) have epic soundtracks in which victories are represented in glorious orchestral crescendos. Survival horror games use industrial soundscapes in which monsters are easily identifiable by dissonant stingers. Such clearly distinguished game musical genres, moreover, are not always even original to the video game medium, but often lean heavily on already existing film musical conventions: the ‘epic’ composing style in fantasy RPGs derives from fantasy film soundtrack idioms, and the minimal, industrial sound of horror games finds a precedent in horror film composition. Because the player has already acquired a (largely unconscious) musical literacy in the interpretation of such audio-visual idioms by watching such films, she will be able to quickly identify what the game music was designed to signify.
The establishment of and immersion in game musical virtual reality thus consists of three overlapping factors: musical affect (A), musical literacy (L) and musical interaction (I). Affect is not used here in the sense of a musical reflection of a personal feeling (like in the Affektenlehre of the baroque) but as ‘an ability to affect and be affected’ (Massumi Reference Massumi, Deleuze and Guattari1987, xvi). The affective connotations evoked by game music help the player interpret the game surroundings and events; her interpretative process is guided by her existing literacy in the idiom of game composing. These two factors help her interact with the game, and as a result also with the adaptive audio, and so the performative process of immersion in game musical virtual reality is perpetually in motion. The three factors in this ‘ALI model’ (van Elferen Reference Van Elferen, Kamp, Summers and Sweeney2016, 34–9) can be visually represented by way of a Borromean knot (Figure 8.1).
Figure 8.1 The ALI model: affect, literacy and interaction.
ALI
The ALI model was originally designed as a tool for the analysis of game musical immersion, but these factors are constitutive of immersion in other types of sonic virtual reality too. Compare, for instance, the musical virtual reality shared by Spotify communities to that shared by a group of MMORPG (massively multiplayer online role-playing game) players. Both processes are operated by the convergence of musical affect, musical literacy and musical interaction, albeit in different ways. In both cases, the affective performativity of musical experience (A) leads to the opening up of individual virtual worlds consisting of memory, emotion and identification. These virtual worlds are supported by musical literacy (L): the acquaintance with audio-visual idioms and genres in the case of game music, and the acquaintance with pop, rock, indie or classical music repertoires in the case of streaming services. The interaction with and through music leads to a further development of the virtual world (I): in games, musical interaction steers adaptive audio, whereas streaming software is explicitly designed to invite interaction with one’s own and other users’ playlists. Moreover, in both cases the fact that this immersive virtual world of music is shared with others often engenders the formation of musical communities that only exist in ‘cyberspace’. The affect, literacy and interaction that characterise game musical immersion thus also enhance the immersion in other types of music-induced virtual reality.
An even more striking example of the relation between different kinds of musical virtual reality is that of video game music fandom. The Chiptune community consists of fans of 8-bit game music who create remixes and new compositions. Chiptune is characterised by the retro timbre and technological limitations of the vintage gaming software. An iconic 8-bit track is Koji Kondo’s 1985 Super Mario theme song, which is the basis of countless Chiptune remixes, covers performed on various instruments and live performances on original Gameboys. Other Chiptune artists create 8-bit versions of non-game-related music, ranging from Vivaldi’s Four Seasons to a-ha’s ‘Take On Me’ and John Williams’s Star Wars theme. Such creations are shared with other video game music fans, discussed on online forums, and appropriated as part of a new canon. Chiptune is a remarkable form of musical virtual reality to the extent that it is a lively online community whose strong social ties have been welded upon participants’ identity by the affective powers of musical experience.
It is not difficult to identify the ALI mechanics of Chiptune’s immersive virtual reality. Interaction (I) probably comes first here: game music fandom is musical interaction extending beyond the confines of actual games, growing from playful navigation into playful creation. Chiptune’s virtual reality hinges on musical affect (A). It is fostered by the lived experience of emotions, connotations and identifications that any fan experienced when they first played the game; fans re-create this virtual world, endow it with the blissful haze of nostalgia, and share their creations as well as their affects with other Chiptune fans. Both dimensions of Chiptune virtual reality are supported by musical literacy (L), as fans recognise the 8-bit sounds and, more importantly, flaunt their knowledge of them by posting their own new creations in online forums. Located simultaneously in the virtual world of game musical play and in that of online communities, Chiptune doubly manifests the immersive power of musical virtual reality.2
Chiptune virtuality also illustrates just how difficult it is to distinguish between musical ‘virtual reality’ and musical ‘real life’. As in the cases of listening to music in the car or wearing your iPod while cycling, creating or listening to Chiptune is very much a part of daily life for those who practise it: not only is the immersion in the playful listening or creative flow woven into daily activities, but this creative activity is also an affective part of fan identity. If the traditional definition of virtual reality describes it as a simulated world outside the ordinary world, the examples of musical virtual reality discussed above suggest that this definition needs to be revised in order to be useful within a musical context.
Individual, Embodied Musical Worlds: Definitions and Approaches
Musical virtual reality is fundamentally different from other forms of virtual reality for two distinct but closely related reasons. First, the same music may evoke greatly differing virtual worlds for each individual listener; second, the virtual reality generated by music is often a physical part of the everyday world. The remainder of this chapter will explore the specificities of musical virtual reality in the light of recent academic debates, and theorise this musical form of reality as governed by the perpetual motion of musical energy.
Immersion and Affect
Musical virtual reality actively engages the listener’s personal imagination. As music in and of itself does not mean anything but only acquires meaning through the associative context of its reception, the virtual reality it calls forth is much less concrete than that established by media such as text, film or gameplay. When two people are immersed in the same video game, it is likely that they have comparable experiences of the game’s virtual reality, guided as they are by clear visual, plot and gameplay information: players of Halo will agree that they are playing a science fiction game with a range of breathtaking intergalactic fights. By contrast, two listeners who are immersed in the same piece of music, say the Halo theme tune, may interpret that music in different ways. An avid Halo player will remember her own gaming experience and share her memories with other fans on YouTube; another listener may not know Halo but recognise the musical idiom of action-adventure, and therefore imagine worlds of fantastic heroism. Yet another listener may enjoy the plainchant and find herself transported into a world of monasteries and contemplative silence. Each of these experiences should be considered a virtual reality – a musically simulated reality – and yet each of them differs from the others in emotional, mnemonic and subjective content.
If such great differences occur, should these listening experiences really be described as virtual reality? If this is virtual reality, does that mean that any programme music – music with the explicit purpose of simulating an experience, worldview, or narrative – generates a virtual reality, too? To put it even more drastically: could that imply that any musical experience can generate virtual reality as listeners are immersed in musically evoked emotions, memories and connotations?
Music’s immersive capacity is such that it can change listeners’ perceptions of time and space. Listening to music can have the effect of a Proustian cake dipped in jasmine tea: any musical element, from the arc of a melody to the timbre of a voice, can evoke the involuntary memory of past times and past feelings (van Elferen Reference van Elferen2011, 31). And that musical associativity is not limited to memory or nostalgia alone. More importantly perhaps, music is able to influence our emotions. Its affective agency can seemingly overrule that of the listener, whose mood can be turned in any direction by music she happens to be hearing. This agency is so strong, indeed, that it can rewrite our experience of everyday spaces, define (sub)cultural identities and support collective narratives (cf. DeNora Reference DeNora2000, 109–26).
Musical immersion has been linked to virtual reality in a number of scholarly contexts. In their study of mobile music, William Carter and Leslie Liu state that musical reality overlays everyday reality as we ‘enter the musical worlds created by songs and albums’ (Reference Carter and Liu2005). Their creative interventions focus specifically on the musical virtual reality enabled by the iPod, exploiting the fact that it allows listeners to ‘drift between the real and the virtual’ (ibid). While Carter and Liu take such musical liminality as a given, Salomé Voegelin’s Sonic Possible Worlds explores the phenomenological processes that establish the conflation of the ordinary world and the virtual worlds. Voegelin relates the phenomenology of musical virtual worlds to literary ‘possible world’ theories. Poised on the threshold of auditory perception, her book argues that deep listening in and of itself represents world-building: ‘[L]istening does not recognize; it listens not for what a sound might represent but hears what it might generate. It hears sound as a verb, as a world creating predicate’ (Reference Voegelin2014, 83).
Voegelin’s thesis resonates with other phenomenologies of music, most notably those of Jonathan Kramer and Jean-Luc Nancy. Both theorists have argued that musical experience alters our perception of the world. Kramer contends that musical time unfolds independently from clock time in a manner very similar to Bergsonian durée, arguing that musical time should be considered as completely separate from – and separating itself from – ordinary time: the time of music has the agency to ‘create, alter, distort, or even destroy time itself, not simply our experience of it’ (Kramer Reference Kramer1988, 5). In a similar vein, Nancy contemplates the ways in which listening creates its own space as well as its own time: ‘The sonorous present is the result of space-time: it spreads through space, or rather it opens up a space that is its own, the very spreading out of its resonance, its expansion and its reverberation’ (Nancy Reference Nancy and Mandell2007, 13). While both Kramer and Nancy attribute world-building powers to musical experience, Voegelin expands such notions into a theory of ‘phenomenological possibilism’ (Voegelin Reference Voegelin2014, 28, 45–8). Countering the transcendent paradigms that dominate previous phenomenological accounts of music, she proposes a sonic immanence in which musical experience is not either virtual or actual, but both virtual and actual (114). Generated by the meeting of sound waves and listening subjects, musical virtual worlds must be considered as part of a ‘continuum between the sonic thing and the sonic subject’ (115): they may emerge at any given time, at any given place, and will differ for any listening individual.
The transformative possibilities of musical experience, according to these phenomenological analyses, are generated by the convergence of musical and subjective factors: the propelling musical forces of melody, harmony, rhythm and repetition initiate a movement of sound and time that is met and complemented by a movement of memory and affect. This interplay between material (sound waves, acoustics, instruments, bodies) and less material agents (thoughts, feelings, memories) can be compared to a ‘Newton’s cradle’ pendulum of five metal balls. A metal sphere on one side swings through the air, passes its energy through the middle three spheres, and makes the sphere on the other side move outward as if by its own accord. This, in turn, sets in motion a self-sustaining perpetuum mobile of kinetic energy. Music, with its remarkable capacity to connect the material with the less material, can be compared to kinetic energy, and its workings to the workings of the pendulum. In a three-dimensional motion across and beyond ordinary time and space, music’s sound waves strike our ears, pass through our neurons, and move memory and emotion as if by an unseen force. This then initiates a chain reaction of further associations and affects. The chain reaction is kept in motion by its own rhythm, its own flow, and thereby its own version of the passing of time – its own durée, as Kramer would say.
While the perpetual motion of musical energy is governed by the conservation of energy, the nature of this energy may vary significantly – it may be kinetic, potential, mnemonic or affective energy; it may be your, my or someone else’s connotations – but the rhythm and flow of the motion will continue. The sway of musical sound waves is passed on to the neurosystem by the organ of Corti in the ear, which translates the physical energy of sound waves to the electro-chemical energy of emotion and connotation. This chain reaction of energies sets musical virtual reality in perpetual motion. Music thus simultaneously affirms and, by generating a virtual reality out of ordinary space and time, defies physical laws: musical energy does not just persist in chronological time nor does it just move in Euclidean space, but it represents a system of interrelated material and less material events. Voegelin’s ‘sonic continuum’ is only possible because of this principle, which applies to musical energy just as it does to kinetic energy. This ‘im/material pendulum’ model is especially appropriate, of course, in the case of video game music, which is explicitly interactive: the less obviously material movement of player associations will feed back directly into (material) adaptive audio by way of gameplay.
The performative capacities of music-induced emotion and connotation were conceptualised as ‘affect’ in the ALI model introduced above. Affect has been an increasing focus of research in arts and humanities since the ‘affective turn’ in critical theory, which had its roots in Brian Massumi’s Reference Massumi2002 monograph Parables for the Virtual. Massumi describes affect as a performative dimension of intensity, an emergent ‘expression event’ that is caused by sensual perception, anchored in embodied subjectivity, and engenders a ‘participation in the virtual’ (Reference Massumi2002, 26–7, 35). He argues that the time and place of affect are necessarily virtual, and therefore always potentially present but not always stable: even if its momentum is not fully actual (which it arguably never can be), affect itself is always virtually and therefore indeterminably present. He describes it as ‘cross-temporal, implying a participation of “temporal contours” in each other, singly or in the looping of refrains’ (34). Massumi adds that this cross-temporality ‘constitutes the movement of experience into the future (and into the past, as memory)’ (34). Read through the framework of the im/material conservation of musical energy, it is clear that the ‘expression event’ of affect is the subjective reaction to the physical encounter with music’s sound waves – a passing on of musical energy. This musico-affective chain reaction leads to musical immersion, which, in Massumi’s terms, is a cross-temporal affective virtuality. Kramer’s, Nancy’s and Voegelin’s phenomenological accounts of the world-building immersion in music are thus embedded in the perpetual energetic interaction between music and affect.
Because of its strong power over the basic perceptions of time and space (the objective axis by which we measure everyday life) as well as over affect and identification (the subjective correlate of that axis), musical experience is able to create a virtual reality. Different from other types of virtual reality, the world that is generated by music is not deliberately created as a specific virtual environment. Even in the case of programme music or video game music, every listener will experience her own, highly subjective and often very detailed Proustian moment. Precisely because of these deeply affective individual differences, musical virtual reality is arguably more immersive than that generated by other media. The answer to the question posed above, therefore, is that indeed any musical experience can generate a virtual world.
Daily Practices and Augmented Reality
Besides entailing significant individual variation, a second way in which musical virtual reality is distinguished from other types of virtual reality is that it is part of rather than parallel to the everyday world. The examples of car audio, mobile music and streaming services suggest that musical virtual reality provides an auditory layer on top of everyday reality. This layer influences our perception of the world: attentively or inattentively immersed in musical virtual reality, we ‘see the world through our ears’. This explains the well-known effect that walking in the city with ‘happy’ music on our mobile music player can make a rainy day seem less gloomy.
This sonic layering of reality has been discussed in game music research in terms of immersion. The bleeding of game sound into the player’s real-life surroundings can lead to domestic irritation, but also to increased player immersion. The 3D sounds of creaking doors and monsters growling in survival horror games create the impression that danger may be lurking in the player’s own house (Cheng Reference Cheng2014, 99–103; Van Elferen Reference Van Elferen2012, 121–7): ‘the sounds and simulations of games’, William Cheng states, ‘resonate well beyond the glowing screen’ (Reference Cheng2014, 14). Moreover, music games such as Guitar Hero, in which the player performs music on haptic interfaces shaped like instruments, demonstrate that this sonic overlap between gaming virtual reality and ordinary reality has emphatically embodied and social dimensions. The inclusion of such visceral components in the virtual reality experience blurs the boundaries between these types of reality even further (Miller Reference Miller2012, 8).
If musical virtual reality is so embedded in daily practices and has such clear physical components, it would seem to exceed the traditional definition of virtual reality as separate from everyday life. Should it then still be considered virtual reality? Since every musician can testify to the embodied nature of immersion in music’s flow (Schroeder Reference Schroeder2006), is not any form of musical engagement a participation in musical virtual reality?
The translation of musical energy into musical affect, as argued above, can establish a musical virtual reality at any time or place. This auditory reality provides affective feedback to the listener’s perception of the day-to-day world. The fact that musical affect has a physical grounding – sound waves, neurotransmitters, chemistry and so on – only increases the strength of musical immersion. Focusing on the corporeal aspects of affect, Melissa Gregg and Greg Seigworth expand Massumi’s theory. They discuss affect as liminal, changeable and multi-faceted, rendering the subject ‘webbed’ through the embodiedness of affective response:
Affect is integral to a body’s perpetual becoming (always becoming otherwise, however subtly, than what it already is), pulled beyond its seeming surface-boundedness by way of its relation to, indeed its composition through, the forces of encounter. With affect, a body is as much outside itself as in itself – webbed in its relations – until ultimately such firm distinctions cease to matter.
Massumi’s ‘participation in the virtual’ is therefore grounded in the physical. Immersed in affect, the body is simultaneously actual and virtual; immersed in musical affect, ‘virtual’ reality and ‘actual’ reality overlap.
The simultaneity of everyday and virtual physicality has also been explored in game music studies. Kiri Miller states that players of Guitar Hero are ‘playing between – that is, playing in the gap between virtual and actual performance’ (Reference Miller2012, 16). For this reason she asserts that the affective and embodied experience of playing a music game is a ‘schizophonic performance’ (15), a form of ‘mending the schizophonic gap, stitching recorded musical sound and performing bodies back together’ (151). In a similar but more explicit manner than in mobile music’s vicarious re-embodiment of sound, the schizophonia discussed at the beginning of this chapter comes full circle in music games. These re-embodiment processes do not, however, eliminate musical virtual reality. While the schizophonia of recording technology foregrounded the virtualising possibilities of musical experience, it did not initiate them: I would argue that any musical experience, whether in performance or in listening, live or recorded, can induce musical virtual reality. Players of music games therefore participate in three coinciding realities: the game’s virtual reality, the music’s virtual reality, and their own physical and social reality.
In addition to the physical suture between gaming virtual reality and real life, playing a music game also has consequences for the experience of liveness and authenticity. Playing these games engenders a simultaneity of various kinds of performance: playing music on a plastic guitar; playing the game by pushing buttons; mimicking one’s favourite artist; and – in a social setting – performing all these things for a social environment. David Roesner argues that the latter performance foregrounds the ‘paratexts’ of that music – its emotional and (sub)cultural connotations (Reference Roesner2011, 283). Every Guitar Hero player will privilege a different aspect of this complex performativity. Some may excel in the musical skills, others will just want to win the game, yet others may want to focus on the social performance of musical paratexts. Although very different in nature, and although all are schizophonic, these performances should be considered ‘live’: they all happen here and now, are embodied, and can be visually as well as aurally perceived (Fritsch and Strötgen Reference Fritsch and Strötgen2012, 58). Schizophonic performance obliterates the intuitive link between acoustic music and authenticity, replacing it with the twin concepts of authentic virtuality and virtual authenticity: a virtuality that is as real as the ordinary world, and an authenticity without tangible grounding. Liveness and authenticity still exist in a mediatised culture, but their properties change.3
Music’s self-sustaining energetic flow is a virtual reality, but not one that matches traditional definitions – as Voegelin states, musical experience is actual and virtual (Reference Voegelin2014, 114), and, as I argued above, it is material and less material. It is separate from the rest of the world, but located in it and caused by physical agents. Rather than as virtual reality, it is more adequately defined as augmented reality, an inclusive concept that has gradually replaced the (practical and theoretical) binary division between real life and virtual reality. Originally used to identify visual and interactive additions to the everyday world – from head-mounted displays to mobile phone apps – augmented reality has extended to the sonic domain. Google’s wearable augmented reality technology gadget Glass, for instance, is fitted with audio technology so that users can simultaneously listen to their favourite streaming station, receive notifications and hear environmental sound. But musical immersion did not need high-tech support for its virtualising capacities: the myriad examples discussed in this chapter illustrate that musical virtual reality is a highly immersive and affectively powerful form of augmented reality that pre-dates the invention of digital technology. The disembodying schizophonia of recording technology may have rendered this sonic augmentation somewhat uncanny, but even if an invisible hand starts the movement of the pendulum’s spheres, it still moves, and its musical energy will still be passed on, and on, and on …
The exciting new research field of musical virtual reality has the potential to challenge existing musicological methodologies and to introduce new approaches. One way to start such disciplinary innovation would be to revisit Carolyn Abbate’s Reference Abbate2004 article ‘Music – Drastic or Gnostic?’ In this influential piece Abbate called for a musicology that researches the performative and less material aspects of playing and listening to music, next to the more traditional hermeneutics of musical scores and historical facts. In her assessment of musicology’s ‘drastic’ new routes, Abbate focused on the performativity of music, exploring the intangibility, ephemerality and even the ineffability of the musical event. Her appeal opened a number of new avenues. Most importantly she invited a critical engagement with music as an event, including such topics as immediacy, presence, liveness. As a result, phenomenological debates about music would receive a new impetus, including questions of dis/embodiment, sensual perception and the materiality of ‘the musical work’.
The study of musical virtual reality would be an excellent point of departure for a renewed drastic musicology. Musical virtual reality is an event, a lived experience, not a work; its manifestation is affective, immediate and interactive; its phenomenology requires renewed debates about the embodiment and perception of music; it questions the existence of the traditional musical work. Musical virtual reality, moreover, also urges questions pertaining to musical epistemology that Abbate only hinted at. If such diverse musical events as visiting a theme park, hearing a film soundtrack, listening to an MP3 player, playing a video game or simply turning on the radio can all engender an affective and corporeal virtual reality experience, then that notion requires far more theorisation than this one chapter can provide. One thing is certain: this new drastic musicology ought to be driven by its subject matter – the perpetual, im/material motion of musical energy.
