What Are Signs?

Semiotics is the study of signs. Charles Sanders Peirce, who was the founder of one of the two main branches of semiotics (the more universal of the branches; the founder of the other, more linguistic branch was Ferdinand de Saussure), described a sign as ‘anything, of whatsoever mode of being, which mediates between an object and an interpretant’.Footnote ¹ The ‘object’ in Peirce’s definition is relatively straightforward; it is a thing that exists, whether as a tangible lump of matter or as an abstract concept. The ‘interpretant’, meanwhile, is the effect of the object on one’s mind – the idea that would typically pop into your mind in relation to that object. The ‘sign’, then, is the intermediary between a thing and an idea, or between the external world and the thought world. In fact, Peirce wrote that ‘every concept and every thought beyond immediate perception is a sign’.Footnote ² Furthermore, signs can be the likeness of the object, like a photograph; they can be an indication towards the object, like an emergency exit sign; or they can be a symbol that only represents the object by convention or by abstract relationship, like a word or the concept of tonality.Footnote ³

Adopting such a broad definition of the term and concept of the ‘sign’ presents us with a massive scope of investigation from the very outset, which the astute semiotician will find profoundly liberating. If anything that represents something to us is a sign, then we are not bound by analogies to language systems or by prescriptive semantics. We can address meanings on the terms in which they are transmitted – describing them with words as humans are wont to do, but not assuming they will map neatly or even consistently to a set of words or concepts. This is quite handy when we look at music, which is a largely non-verbal and abstract medium (even singers can convey abstract and non-verbal meanings, like emotions, alongside the words being sung).Footnote ⁴ It becomes especially useful when we look at music in video games, where music can be directly associated with events, characters and areas, even while maintaining some level of abstraction from these things.

How Signs Are Used to Communicate

However, semiotics is not necessarily the study of communication. Jean-Jacques Nattiez, who developed a Peircean semiotics for music, observed that while communication through music is possible, it cannot be guaranteed.Footnote ⁵ Communication depends on a meaning being inscribed by one person, then a corresponding meaning being interpreted by another person. It is possible for the other person to interpret a meaning that is completely different to the meaning inscribed by the first person – we might call this a mis-communication.Footnote ⁶ And music is generally very open to interpretation.

Music cannot prescribe its own meanings, except within cultural and textual conventions and relationships. It can ‘connote’ meanings by implying relationships outside of itself – evoking emotions like the sadness we feel in Beethoven’s ‘Moonlight Sonata’,Footnote ⁷ or suggesting scenes like the flowing waters of Smetana’s VltavaFootnote ⁸ – but it does not typically have a mechanism to ‘denote’ or exactly specify meanings. Any denotations we hear in music (and even many of its connotations) are contingent on external relationships. For example, Smetana’s Vltava may suggest to us the movement of a grand river, but to do so it relies on our familiarity with Western musical tropes (such as the relationship between conjunct motion and the concept of ‘flow’); to link the music to the eponymous Czech river itself requires the composer’s direct explication in the title of the work.

Like all human creative output, music exists within contexts that provide such conventions, and it moves between contexts as it is encountered by people, in the process forming relationships between contexts. Even musical analysis (sometimes described as an investigation of ‘the music itself’) is what Theo van Leeuwen calls a ‘recontextualization’; meanings tend to attach themselves to works as they are experienced and distributed, even (or especially?) when this is done with an analytical purpose.Footnote ⁹ The ever-present existence and never-ending negotiation of context allows music to be denotative, within specific bounds. For van Leeuwen, ‘musical systems (systems of melody, harmony, rhythm, timbre, etc.) do have “independent” meaning, in the sense that they constitute meaning potentials which specify what kinds of things can be “said”. … Meaning potentials delimit what can be said, the social context what will be said’.Footnote ¹⁰ So when we start playing a video game and hear its music, we are not accidentally listening to some tunes with no meaningful relationship to the images we see or the interactions we perform. Each element of a video game has an influence over the meanings we interpret in the other elements, resulting in a cohesive and semiotically rich experience filled with the potential, at least, for communication.

For a ‘meaning potential’ to turn into a ‘meaning’, the potential just needs to be fulfilled through experience, much like how specific decisions can affect the final stages of some games, giving you one ending from a set of possible endings. For a ‘potential communication’ to turn into a ‘communication’, however, there needs to be a correlation between the composer’s intended meanings and the meaning suggested by the interpretants in the player’s mind. This is a minefield. Game composers and developers can try to influence the interpretants that music will form in the player’s mind by creating specific contexts for the music, such as by co-ordinating or juxtaposing music with other game elements. However, there are other factors that affect how the player’s mind works (everything from their past gaming experience, to their personality, to what they had for dinner). Every experience the player has during gameplay has a meaning, but these meanings form part of a negotiation of communication between the player and the game – a negotiation that involves sounds, images, movements and inputs. Musical signs form part of this negotiation, and their meanings can depend as much on the player themselves as on the music’s initial composition. Tracking down the source of musical meanings, and tracing how these meanings move through and influence gameplay, are key to understanding game music’s signs and communications.

Two Semiotic Domains

Next time you play a game, take notice of the things you are observing. Take notice of the characters: how they are dressed, how they move, how they interact with you. Take notice of the scenery: its colours, shapes and textures; its architecture, time period, planet of origin or state of repair. Take notice of the sounds of the concrete jungle, or the living forest, or the competitors’ race cars or the zombie horde. Take notice of the music: the shape of its tune, the texture of its harmonies and rhythms, how it alerts you to characters or objects, how its ebbs and flows guide your emotional responses. The video game is providing you with a constant stream of signs that signify many different things in many different ways; your entire perception of the gameworld is constructed by communications through visual, aural and/or textural signs.

While you are at it, take notice of the things that you are communicating to the game. You are making inputs through a keyboard, mouse, controller, joystick or motion detector. Your inputs may be directional (‘move my character to the left’), functional (‘change up a gear’) or metafunctional (‘open the inventory menu’). Many of your inputs are reactions, prompted by conscious needs (‘I need to see what is in my inventory’) or by semi-conscious or unconscious needs (flinching when you hear a monster right behind your character). Many are the result of your choices, whether to move towards a strange sound, to cast a particular spell or to turn up the music volume. Your actions are providing the game program with a stream of input signs that come from you and that represent how you want to interact with the gameworld; your interactions are constructed of signs that could take the form of keystrokes, button presses, movements, keywords, voice commands, configuration file changes and so on.

The game music’s communications to you, and your communications with the game and its music, come from two separate semiotic domains that have distinct (though related) sets of semiotic activity. The first semiotic domain is centred around the creation of the video game and the composition of the music: this is when musical (and other) elements of the game are put into the video game program based on creative and developmental choices. The second semiotic domain is centred on your actions in gameplay: this is when the musical (and other) elements of the game program are actualized into your experience of the game through your interactions. When you are playing a game, you are hearing musical signs from both semiotic domains – in the most basic sense, you hear the composer’s original musical signs (initial composition) in the way they are being configured by your interactions (gameplay). And while it often makes sense to look at these together (since games are made to be played), there is some valuable information about both the game and how you play it, and the music and how you hear it, that can be obtained from looking at them separately.

Initial Composition and Initial Meanings

There are two kinds of signs that are put into video game music when it is composed and put into the game program: musical signs and configurative signs. Musical signs are the things that we typically think of as music, like notes, melody, harmony, instrumentation, dynamics and so on. These typically function the same way in video games as they could be expected to in any other context. For example, if you remove a music file from a video game’s program folder and play it in an audio player, it will often sound like regular non-interactive music.Footnote ¹¹ You can also analyse these signs much as you would any other piece of music. Accordingly, these are often the first signs to be analysed when we look at video game music. These are the signs that bear most resemblance to film music, that are transposed into orchestral settings, that are played on the radio, and that you buy as part of an Official Soundtrack album. And, if adequately accounting for the cultural contexts of both composition and analysis, investigating the musical signs as at initial composition can reveal a lot about the music and its role within the game.

Configurative signs, meanwhile, are the signs that turn music into video game music. These are things that come into the music through the functions and designs of the video game program, and can include timing, repetition, layering and responses to interactivity. They take musical signs and configure them to work as part of the game. For example, the video game Halo 3 (2007) plays music as the player enters certain spaces filled with enemies. The timing of the start of the music, and the consequent correspondence between the music and the newly entered space, are configurative signs. Like earlier games in the Halo series, Halo 3 also turns off the music after the player has spent too long in the one space to encourage them along and to minimize repetition, and this is also a configurative sign.Footnote ¹² Configurative signs contextualize musical signs within the game program by determining how the music will fit in with the other game elements. They are, therefore, very important in determining how (and how effectively) music can denote meanings within the game. The musical signs in Halo 3’s soundtrack may independently connote a sense of action, but the configurative signs that line up the start of the music with entering a theatre of action are what allow the music to denote the experience of battle.

Unlike musical signs, configurative signs are hard to remove from the video game and analyse separately. The functions built into the game program can be difficult to observe independently without access to the video game source code and the requisite knowledge to interpret it. Likewise, the precise mechanisms used by audio subsystems or middleware require a certain amount of technical expertise to both implement and interpret. However, we can make educated deductions about these functions based on gameplay experience. Battles in Halo 3 tend to take place in open spaces that are joined by corridors, roads or canyons. After entering several of the open spaces and hearing the music start, we can infer that the game contains a function to start music when entering open spaces. Further gameplay can confirm or qualify the hypothesis, and then analysis can focus on when and why these configurative signs are implemented in the game and what effect they have on the music. This must be done with a little care, however, as there are additional semiotic processes taking place during gameplay.

Gameplay and the Player’s Influence

Gameplay simultaneously changes nothing and everything about video game music. Just as in the semiotic domain of initial composition, music within gameplay consists of musical signs and configurative signs. These are, for the most part, the same sets of signs that were present at initial composition. However, during gameplay, the player’s interactivity actuates the configurative signs, which then act on the musical signs to determine what music the player hears and how the player hears it. So, while the Halo 3 game program contains both the music cues to play in certain areas and the program functions to make the music stop playing after a specific amount of time to hurry the player along, it is only the player’s choice to linger that cements the cessation of music in that area as a fact of the musical experience. If the player chose to clear the area and move on quickly, they would hear a transition to another music cue, but the cessation of music in that area would not be part of their gameplay experience. The player’s choices add and exclude musical potentials in the played experience (an important thing to remember when analysing music from within gameplay); they can activate a different set of configurative signs than another player and end up with a subtly but observably different experience.

However, the player brings more to the game experience than just the power to activate configurative signs. They bring a set of signs that correspond to them personally. These can include the choices they make and the reasons they make them, their musical, gameplay or stylistic preferences, their mood, their cognitive associations or even artefacts of the world around them. These personal signs strongly influence which of the configurative signs built into the game the player will activate during gameplay. The activated configurative signs subsequently act on the musical signs of the game, crystallizing a musical experience from the potentials that were built into the game mixed with the effects of the player’s personal experiences. One of the simplest examples of the influence of personal signs is the way your mood affects how you drive in a racing game. If you are having a bad day, you are more likely to drive rashly, make errors, spin out and lose the race. In this circumstance, not hearing the victory music at the end of the race is a result of the negative signs floating around in your mind being expressed in your interactions.

Figure 13.1 demonstrates how each set of signs relates to other sets of signs within the semiotic domains of initial composition and gameplay; an arrow from X to Y in this diagram indicates an ‘X acts on Y’ relationship. The arrow from personal signs to musical signs may be unexpected; it indicates that there are circumstances in which the player can add musical signs directly to the game (such as by playing their own music instead of the soundtrack, or by providing a music file to be played within and as part of the game like in Audiosurf, 2008) or in which the player’s perception of certain musical signs is strongly influenced by their personal experience (such as dancing to ‘I Don’t Want to Set the World on Fire’ by the Ink Spots at their wedding, then hearing the song played in association with a post-apocalyptic wasteland in Fallout 3, 2008).

Figure 13.1 Actions of signs in the semiotic domains of interactive configuration and gameplay

Because the musical experience of gameplay is constructed from a combination of signs from the music’s initial composition and signs from the player themselves, every player’s experience of a game’s music is unique. Of course, all experiences of a game are similar, since they are built from the same musical and configurative potentials that were originally put into the game. However, it is important to be mindful that approaching video game music through gameplay changes the semiotic domain in which listening and analysis occur, and that there are more sources of signs in the semiotic domain of gameplay than just ‘the text itself’. This is one of the great differences between video games and other texts, such as films; while any viewer of a film brings a lot of their own thoughts, opinions and signs to the task of interpreting a film, they (usually) do not change the film itself in the process of viewing it. Espen Aarseth describes texts like video games as ‘ergodic’, meaning that work has to be done to them in order to interpret them.Footnote ¹³ Video games and their music are not the only kinds of ergodic texts, but they are arguably the most widespread (or, at least, commercially successful) examples in today’s world. Furthermore, an interesting corollary of doing work to a text is that, as Hanna Wirman points out, the player can be considered as an author of the text (especially when considering the ‘text’ to be the played experience of the game).Footnote ¹⁴ So video game music is simultaneously just ‘regular old music’, and a relatively ‘new’ kind of text with potentials for expression and creativity that we have barely started to explore. The following case studies of musical examples from the Elder Scrolls games demonstrate both aspects of this duality.

Linking Games Through Musical Composition: Elder Scrolls Title Music

The scores for the Elder Scrolls games maintain an aesthetic congruence with the story world of each game. For example, Skyrim’s score frequently makes use of a male choir to invoke an aesthetic aligned with modern imaginations of Vikings, an overt expression of Scandinavian inspirations that are also built into the game’s musical foundations in references to Scandinavian nationalist symphonic music.Footnote ¹⁵ However, the scores of each game also maintain congruence with the scores of the other games using variations on several shared melodic and rhythmic themes, creating the potential for familiarity. This is a powerful mechanism for drawing the player into a video game series, and the Elder Scrolls games begin this work even before gameplay proper, as the title themes of each game exhibit some of the clearest thematic parallels. When a player plays Oblivion for several hundred hours, for instance, and then begins to play Morrowind because they want to experience the earlier game’s story as well, there are musical signs in Morrowind that suggest they are in the same gameworld that they enjoyed in Oblivion.

The title music from Morrowind uses a melodic theme that can be separated into three melody fragments, marked A, B and C in Example 13.1. Fragment A initiates a sense of action by moving quickly from the tonic to an emphasized minor third (a note which, as tonic of the relative major, suggests optimism in this context), and then repeating the action with further upward progress. The roundabout return to the tonic is like heading home after a mission with a momentary detour to pick a flower that you need to make a potion. Each ‘action’ in Fragment A is made up of a rhythmic figure of a minim on the first beat of the bar preceded by two quavers (which may be counted out as ‘three-and-one’) – a figure that occurs frequently in this theme and features prominently in the themes of subsequent games. Fragment B repeats the first ‘actions’ of Fragment A but then achieves and affirms the higher tonic. Fragment C triumphantly pushes on upwards to the relative major’s higher tonic – the fulfilment of the optimism of Fragment A – and then descends progressively to where this journey all began. Morrowind’s title music repeats this sequence three times, each with increasing volume and orchestral richness, each time attempting to draw the player deeper into a sense of adventure with which to better enjoy the imminent gameplay.

Example 13.1 Excerpt of title music melody from Morrowind

The primary melodic theme from Oblivion is a variation on the first two parts of Morrowind’s theme (each variation marked in Example 13.2 as A2 and B2). There is a marked increase in the pace of Oblivion’s version of the theme, which is somewhat disguised in this transcription by the shift to 6/8 time, a shift which results in an aesthetic more akin to that of a mainstream pirate film than of a high fantasy epic. The ‘three-and-one’ rhythmic figure is carried over, though here as ‘three-and-four/six-and-one’ figures. The additional pace of this theme gives the figures a somewhat more adventurous feel, bringing to mind swordplay or the sound of a horse’s galloping hooves. Notably, the primary melodic theme of Oblivion does not use Morrowind’s Fragment C at all – following Fragment B2, the title music instead enters a series of dramatic variations on Fragments A2 and B2. However, Fragment C does make a (somewhat veiled) appearance later in Oblivion’s title theme. Though Fragment C3 in Example 13.3 is a very simplified version of Fragment C, its rhythm and downward shape are both suggestive of the Morrowind fragment and fulfil a similar homeward purpose. The return to the tonic is, however, attained with much less certainty. With the benefit of hindsight, this could be seen as an allegory for the uneasy peace achieved at the resolution of Oblivion’s main quest line, which is revealed in Skyrim to have been very much temporary.

Example 13.2 Excerpt of title music melody from Oblivion

Example 13.3 Later excerpt of title music melody from Oblivion

Skyrim’s primary melodic theme is varied even more substantially from the original Morrowind material than was Oblivion’s primary theme, to the point that it may be more accurate to describe Skyrim’s primary melodic theme as a variation on Oblivion’s theme rather than that of Morrowind. It continues and expounds upon some of the ideas behind Oblivion’s variations, such as pacing, a 6/8 time signature and an air of adventurousness. Like that of Oblivion, the Skyrim title music initially uses only variations of Fragments A and B (marked as A4 and B4 respectively in Example 13.4), which it then repeats with slight variations (Fragments A4* and B4* in Example 13.4). The theme as a whole also has a much smaller pitch range than earlier themes, though it makes up for this in loudness. The main element that links Skyrim’s primary melodic theme back to Morrowind is the rhythmic figures or ‘actions’, as these are again featured prominently. It may even be argued that, at this stage, the rhythmic figures are the most significant shared elements tying the title music of Morrowind, Oblivion and Skyrim together, and the main musical sign to the player of commonality between the games (prior to the start of gameplay, at least).

Example 13.4 Excerpt of title music melody from Skyrim

However, later in Skyrim’s theme the original Morrowind theme returns in an almost pure state. Example 13.5 shows an excerpt from Skyrim’s title music including the two most prominent vocal parts and a trumpet accompaniment. The three original Fragments from Morrowind are reproduced very closely in Fragments A5, B5 and C5, most prominently in the lower vocal part and the trumpet line. This is arguably more of a choral and orchestral arrangement of the Morrowind theme than a variation upon it.

Example 13.5 Later excerpt of title music melody from Skyrim (with trumpet accompaniment)

From a semiotic point of view, the melody that I have called the Morrowind theme above is a sign, as are the Fragments A, B and C that constitute the theme, and even the rhythmic figures or ‘actions’ that recur throughout the theme and its variations. All of these fragments are capable of bearing meaning, both as small elements (sometimes called ‘musemes’)Footnote ¹⁶ and combined into larger structures. At the time of Morrowind’s release, these were symbols of the game Morrowind and, after gameplay, of the experience of playing Morrowind. Through the variations on the theme presented in the scores of the subsequent games, they have been extended into symbols of the experience of playing an Elder Scrolls game. The particular instances of the theme still refer to the game in which that instance is found (that is, Skyrim’s versions of the theme as sung by a ‘Nordic’ choir are still very much symbols of Skyrim), but they also refer laterally to the sequels and/or prequels of that game and to the series as a whole.

The latter example is a good illustration of this twofold signification, as it is functionally a part of the Skyrim theme but is also clearly derived from Morrowind. Its inclusion is, in fact, rather fascinating. The excerpt in Example 13.5 begins at 2 minutes 8 seconds into Skyrim’s title music, but Skyrim’s title menu is remarkably simple, presenting new players with only three options (start a new game, view the credits or quit). It is entirely possible that a first-time player could spend less than 20 seconds in the menu before starting a game and so could miss the close reproduction of the Morrowind theme entirely. Its inclusion allows Skyrim’s title music to powerfully signify both nostalgia for past games and the novelty of a fresh variation to long-term fans of the series,Footnote ¹⁷ while its delayed start prevents it from interfering with the title music’s essential function of priming seasoned Elder Scrolls players and neophytes alike for the imminent quest through established musical tropes of adventure and place.Footnote ¹⁸ Different groups of people may experience a different balance of nostalgia, novelty and expectation, which is both testament to the richness of the score’s weave of musical and configurative signs, and an example of how the semiotic content of music can be altered by the semiotic processes of gameplay.

Linking Experiences Through Musical Gameplay: Skyrim’s Action Music

Playing Skyrim takes you on many journeys. Some of these journeys give you a chance to discover new parts of the gameworld, new sights and sounds and characters and challenges. Other journeys are strangely similar to journeys past (all dungeons start to look the same after a while). The journeys are all somewhat alike, of course, on account of being made of the same basic elements: the parts that make up the game Skyrim. Building familiarity with gameplay mechanisms and signifiers early in gameplay can help the player engage with the game’s world and story. As long as familiarity avoids crossing the line into boredom, it can be a useful tool in the player’s arsenal for success. In a game like Skyrim, success can be achieved in large and small ways: finishing quests (both major and minor) and winning battles.

The data files of Skyrim contain eight full-length (between one- and two-minute) cues of combat music that are played during battles (there are also some derivative and/or smaller versions that are not considered here). These cues share several commonalities, including fast tempi, minor keys, high-pitch repeated staccato or accented notes and dramatic percussion that emphasizes the first beat of each bar – traits that align with the typical role of combat music, which is to increase the sense of action and tension in the sonic environment during combat scenarios.Footnote ¹⁹ Skyrim and its prequels are open-world games and the player can enter combat scenarios inadvertently; the pleasant music that accompanies exploration of the countryside can be suddenly interrupted by combat music, and this is sometimes (if the enemy has not yet been seen or heard) the first thing that alerts the player to danger. Helpfully, Skyrim’s combat music has a sign at the start of each combat music cue to point out what is happening. Figure 13.2 shows waveforms of the first five seconds of each of the main combat music cues. Each cue has an abrupt start that is, in most cases, louder than the ensuing music. Loud strikes on deep drums emphasize the shift from bucolic exploration to dangerous combat and prompt the player to respond accordingly.

Figure 13.2 Graphical representation in Audacity of waveforms of combat music from Skyrim (mixed down to mono for simplicity; cues listed by filename with Original Game Soundtrack titles indicated) [author’s screenshot]

The first time a player encounters combat music in Skyrim is during the introductory scenes of the game, when the player’s newly minted character has their head on the executioner’s block in the town of Helgen. Proceedings are interrupted by the appearance of the dragon Alduin, the prophesied consumer of the world, who flies in and attacks the town. The musical accompaniment to this world-changing event is the cue ‘mus_combat_04’ (named ‘Blood and Steel’ on the Original Game Soundtrack).Footnote ²⁰ Immediately preceding the initial crash of percussion in the music is Alduin’s ominous roar and visual appearance (see Figure 13.3). The sign linking combat music and danger could scarcely be more obvious.

Figure 13.3 Alduin’s appearance at Helgen (silhouette between tower and mountain), and moment of first combat music in Skyrim [author’s screenshot]

Having been introduced to combat music in this way, the player can more easily recognize combat scenarios later in Skyrim. Down the road from Helgen, the player may come across a wild wolf and once again hear combat music. A wolf is a smaller threat than a dragon, though likely still a test of the new character’s mettle. On hearing the initial blast of drums, the player can whip out their war axe or sword and take care of the situation. Yet, as previously mentioned, Skyrim is an open-world game and threats may come from any direction. The next wolf to attack may come from behind a tree (the music begins when an enemy becomes aware of the character’s presence, not the other way around), so the music may be the first sign of danger. It can operate in this way because it was initialized as denoting threats from the very start.

As the player learns more about how gameplay in Skyrim works, they may choose a play style that emphasizes strategic combat over brawn. The skill of ‘sneaking’ involves both moving quietly and staying out of sight in order to move close enough to an enemy to strike a critical blow, ideally without the enemy becoming aware of the character’s presence until it is too late. The player has learned that combat music starts when an enemy becomes aware of the character’s presence and starts to attack, so the absence of combat music becomes the sign of successful sneaking. Combat music during an attempt to sneak up on an enemy indicates to the player that the game is up – they have to run or fight, and quickly. Sneaking is a semiotic negotiation, wherein the player is trying to convey the right combination of signs to the game to get the desired combination of signs in return. It depends both on the game initializing combat music as a sign of danger, and on the player learning and subsequently manipulating the game’s signs for their own ends.