2.1. Multimodality

One premise for an embodied perspective on music cognition is that human perception is inherently multimodal in nature, meaning that all of our senses and modalities mutually influence one another. In fact, multimodality ought to be seen as the norm rather than a deviation in human perception in general (Berthoz Reference Berthoz1997). A small thought experiment will illustrate this point: imagine that you hear the sound of a glass crashing to the floor behind you. You will most likely turn to see what has happened. Already, then, three modalities are at work: the auditory, the visual and the vestibular. Your sense of hearing detects the sound of the breaking glass and guides your turning so that your directionally limited sense of sight can identify the source of the sound, while your vestibular modality keeps you balanced and informs you about the orientation and movement of your head and body.

There are also times when our multimodal capacities can generate cognitive conflicts, as perhaps most famously demonstrated by the McGurk effect. In this illusion, the auditory component of one sound is paired with the visual component of another sound, leading in some cases to the perception of a third sound (McGurk and MacDonald Reference McGurk and MacDonald1976). Similar types of audio-visual integration have been found in music (Vines, Krumhansl, Wanderley and Levitin Reference Vines, Krumhansl, Wanderley and Levitin2005), and experiments have shown that such audio-visual integration seems to occur pre-attentively (Thompson and Russo Reference Thompson and Russo2006). These findings indicate that the perception of body movement can, for better and for worse, influence how we experience sounds.

2.2. Objects and actions

The action–sound approach to music technology practice presented in this paper derives from an acknowledgement of the powerful connections between sounding objects, sound-producing actions and the resultant sounds that exist in nature. We can define such connections as action–sound couplings. The human capacity to imagine and anticipate sounds seems to be based on our knowledge of the acoustical features of sounding objects and the mechanical properties of sound-producing actions. Schematically, as illustrated in Figure 1, a sound is produced by two (or more) sounding objects acting upon one another, or what might be called an object–action–object system. This can be anything from the direct action of hitting a drum with one's hand to the complex mechanical interaction that happens when a finger presses a piano key. The final sound is based on the acoustic properties (size, shape and material) of each of the objects involved in the interaction, and the mechanical laws of the actions that work upon those objects (external forces and gravitational pull).

Figure 1 Sketch of the main elements necessary to produce sounds: Two sounding objects and one sound-producing action

Our life-long experience with the acoustic and mechanical properties of objects and actions allows us to predict the sound of an object–action–object system even before it is heard. This can be seen in a variation upon the thought experiment of the broken glass: imagine that you see a glass being pushed off a table in front of you. Even before the glass hits the floor, you will construct an expectation regarding what will happen both sonically and visually. This is to say that seeing (or imagining seeing) a glass falling toward the floor is enough to generate expectations regarding the timbral qualities and loudness of the sound that will result.

Studies of sound-source perception have shown that it is also possible to obtain a lot of information about objects and actions only by hearing a sound (Giordano Reference Giordano2005). This information might include anything from the shape, size and material of the objects involved to the actions working upon them, such as whether the object was dropped or thrown (Gaver Reference Gaver1993a, Reference Gaver1993b). The human ability to identify the properties of objects based solely on sound seems to be surprisingly reliable and accurate (Carello, Wagman and Turvey Reference Carello, Wagman and Turvey2005), even in everyday perception of impact sounds and sources (Rocchesso and Fontana Reference Rocchesso and Fontana2003).

2.3. Action–sound types

By combining terminology from Schaeffer (Reference Schaeffer1966) and Cadoz (Reference Cadoz1988), we may identify three different action–sound types, as presented by Godøy (Reference Godøy2006):

• • Impulsive: The excitation is based on a discontinuous energy transfer, resulting in a rapid sonic attack with a decaying resonance. This is typical of percussion, keyboard and plucked instruments.

• • Sustained: The excitation is based on a continuous energy transfer, resulting in a continuously changing sound. This is typical of wind and bowed string instruments.

• • Iterative: The excitation is based on a series of rapid and discontinuous energy transfers, resulting in sounds with a series of successive attacks that are so rapid that they tend to fuse. This is typical of some percussion instruments, but also tremolo effects on other instruments.

The aim of categorising sound-producing actions into these three action–sound types is not to classify the instruments as such but to suggest that the mode of the excitation is directly reflected in the corresponding sound. There are, however, several possible combinations as well. A violin, for example, may be played with a number of different sound-producing actions, ranging from impulsive pizzicato to sustained legato.

As sketched in Figure 2, each of the action–sound types may be identified by the energy profiles of either the action or the sound. Note that two action possibilities are sketched for the iterative action–sound type, since iterative sounds may be the result of either the construction of the instrument or the action with which the instrument is played. An example of an iterative sound produced by a continuous action can be found in a cabasa, where the construction of the instrument produces the iterative sound. Playing a tremolo on a piano, on the other hand, involves a series of iterative actions, even though these rapid actions tend to fuse into one superordinate action (Godøy, Jensenius and Nymoen Reference Godøy, Jensenius and Nymoen2010).

Figure 2 Sketch of action energy and sound levels for the three main types of sound-producing actions. The dotted lines/boxes suggest the duration of contact during excitation. Note that iterative sounds may be the result of different types of action

2.4. Action–sound separation

In addition to the different types of relationships between actions and sounds mentioned above, we may also distinguish between how closely the action is connected to the sound. Thelle (Reference Thelle2010) has distinguished between five levels of action–sound separation:

• • Incorporated: The human body is itself the sound-producing element, as when one sings.

• • Direct: The performer is in direct contact with the sound-producing element, as when one plays a string instrument with one's fingers.

• • Mechanical: There is one (or more) physical layer(s) between the body of the performer and the sound-producing element, such as the sticks or mallets of a percussionist or the key action mechanism of a piano.

• • Analogue electronic: The body controls electronic circuits through an analogue chain, as when one plays an analogue synthesiser.

• • Digital electronic: The body triggers streams of digital data that are connected to a sound engine through one or more mapping layers, as when one works with software instruments on a laptop.

Here, the separation between action and sound goes from being totally embodied in incorporated instruments to being largely disembodied in digital instruments. In practice, however, there are often many possible combinations of separations. For example, string players may be in direct contact with their instrument using the left hand, while playing it with a tool (a pick or a bow) with the right hand. There are also several different levels of interaction with analogue electronic instruments, from the immediate interaction with the electric current of the Victorian synthesiser (Collins Reference Collins2006) to the mechanical manipulation of keys and buttons of analogue synthesisers. Still, it is useful to clarify that we are talking about a continuum between interacting very closely with the sound-producing element itself to interacting with one or more ‘layers’, either physical or virtual, between the user and the element.

2.5. Couplings and relationships

In the following discussion, the term action–sound coupling will be used to describe the three first levels of separation; that is, incorporated, direct and mechanical. The connections between action and sound found in electronic instruments (both analogue and digital) will be identified as action–sound relationships. Note that these two terms (couplings and relationships) derive from the nature of the connections between the objects and the actions, not from our perception of them. In other words, a tone played on an acoustic piano is based on an action–sound coupling, while a tone played on a digital piano is based on an action–sound relationship. This is the case even though we may believe that the tone from the digital piano is as ‘natural’ as that from the acoustic piano. However, no matter how well the action–sound relationship of a digital piano is designed and constructed, it will fail the moment the power is turned off. For this reason, it is also useful to clarify that action–sound couplings are based on mechanical laws, while the action–sound relationships found in electronic instruments are designed and constructed electronically.

Our knowledge of action–sound couplings continuously develops as we experience new couplings throughout our lives. Similarly, as we surround ourselves with an increasing amount of technology, our knowledge of different types of electronic action–sound relationships is also in continuous development. Today, electronic action–sound relationships can be found in door-bells, mobile phones, musical instruments, TVs and computers. These relationships may be strong or weak, direct or indirect, but they still influence our experience of the devices with which we interact.

Even though electronic action–sound relationships may become so familiar that they start to feel natural, it is questionable whether our perception of them may ever be as strong as that of a coupling. For example, even though we may have always heard the same type of piano sound when we pressed the key on a digital piano, we still cannot be absolutely certain that this will continue to be so. One day, there may be no sound at all because the power is out, or there may be another sound because someone changed the settings on the instrument. If this happens, we may be surprised, but we will understand that this is not an impossible outcome, given that we are dealing with an electronic instrument. On the other hand, we simply know that we will never hear nothing when we pluck the string of an acoustic guitar (or hit the key on a piano). Such outcomes are in fact impossible in acoustic instruments, because they are based on mechanical and acoustical laws.

2.6. Affordance

Within his ecological psychology, Gibson (Reference Gibson1977) introduced the term affordance to represent the action possibilities of a given object in a given environment. For example, a chair affords sitting down, but it also affords acting as a table, being thrown at someone or being used as a percussion instrument. Within the context of product and usability design, Norman (Reference Norman1990) developed his own take on the concept of affordance by suggesting that an object's affordance denotes the interaction possibilities that a user would typically identify with it. While one might argue that some affordances are more basic than others, most affordances are probably learned through simply living. We continuously expand our knowledge about affordances through our daily interaction with objects in the world. For example, though we probably first learn that a chair is made for sitting, we encounter new affordances for it all the time.

From a musical point of view, we might say that an acoustic guitar affords the sound of vibrating and resonating strings, and we can tell this simply by looking at the construction of the instrument. The opposite is usually the case when we encounter electronic instruments, however. Keyboard-based synthesisers, for example, have an interface (keys) that affords piano-like actions that are impulsive in nature but that are often used to control any type of sound model, including sustained and iterative sounds as well. The keyboard interface, then, suits the production of impulsive sound types, but the production of sustained sounds, such as strings or wind instruments, may make the instrument feel ‘unnatural’ and ‘inexpressive’ to both the performer and the perceiver.

2.7. Action–sound palette

Following the idea that an object can have multiple affordances, we can also imagine an action–sound palette of different sounds emerging from an interaction with a sounding object. Referring back to the imagined falling glass, what if it were a plastic cup instead? Then the result of contact with the floor would probably be the cup bouncing off with a ‘plastic-like’ sound rather than smashing into pieces. If we had assumed that the vessel in question were made of glass, we would probably be surprised by a plastic-like sound, but it would still be a possible outcome, if we were not absolutely certain about its composition. However, if we heard, for example, a baby's cry when the glass hit the floor, this would disrupt the the laws of nature entirely, and we would place the cry somewhere other than in the kitchen with the fallen glass. This is because a baby's cry is not part of the action–sound palette of the object–action–object system in use. Therefore, the action–sound palette is important to our perception of the relationships in a given system, and it is most likely deeply rooted in our process of cognition.

For couplings, an action–sound palette is restricted to the possible combinations of material and action properties. For example, the sound of a glass breaking will depend on the material, size and shape of the glass, as well as its speed at the moment it crashes into the floor. These parameters may vary, but only within a fairly narrow palette. The possible action–sound relationships of an electronic door bell, on the other hand, are potentially infinite, and the sound may range from ‘ding-dong’ or ‘beep’ to an excerpt from a piece of music. In this case, we might perceive a ‘ding-dong’ sound to be more natural than a ‘beep’, because it inherits some of the action–sound qualities of a mechanical door bell, upon which the design of the electronic door bell may have been modelled. However, if music starts playing when we press the button, we will tend to experience a rather unnatural or ‘weak’ action–sound relationship. A large action–sound palette can therefore be problematic in some contexts, but it can also inspire great creativity in others. For example, experimental composers and performers often try to extend the palette of their instruments in various ways. This can be done acoustically, by, for example, ‘preparing’ the instrument through the addition of various types of mechanical parts, such as coins, paper clips, steel wool and so on. It can also be done electronically by, for example, applying various types of sound effects. In some cases, such action–sound palette extensions will become part of the standard action–sound repertoire, such as the use of distortion and wah-wah pedals with electric guitars. Even though such effects originally represented drastic changes to the sonic result of a guitar, there are few people who would find such guitar effects perceptually challenging today.

2.8. The action–sound approach

To summarise, the action–sound approach is based on preserving an awareness of the basic laws of acoustic instruments, and our cognition of musical sound, when developing electronic instruments. This is not to say that we should only make electronic instruments that mimic acoustic instruments. Instead, we should simply begin the design and building process of electronic instruments with the possibilities of the laws of nature and our human cognitive capacities. Starting with such a perspective will also make it easier to explore how departing from these laws and capacities may allow for new and interesting musical results.

3.1. Acoustic 1: everyday sonic objects

The opening lecture is concerned with sound in general, the theory of sound production and the basics of sound perception. Students are asked to bring with them a sounding object of any sort at all, including, for example, a coffee cup, toothbrush or balloon. Their task is to explore the sonic possibilities of their objects, both with and without tools, and to create a little piece to play for the others. These pieces are used as the starting point to introduce the three main action–sound types (impulsive, sustained, iterative), and to discuss the action–sound palettes of their objects.

3.2. Acoustic 2: amplification

The second lecture is focused on how amplification can be used to increase and alter the acoustic sound of the objects they chose for the first lecture. We have available a collection of amplifiers/speakers of different sizes (ranging from small PC speakers and miniature guitar amplifiers to studio monitors) and microphones (both cheap and expensive dynamic, condenser and contact microphones). Students are introduced to the theory behind how microphones and speakers work, and how the different types of equipment will influence the final sound. They are then free to explore sonically how the different speakers and microphones influence the sound of their sonic objects (Figure 4).

Figure 4 Picture of a group of students testing out different types of microphones and speakers

3.3. Acoustic 3: sound effects

The third lecture is devoted to how sound effects can change the sound of the sounding object. Here, students are introduced to the basics of sound processing (delay, filtering, equalising), and they are given the opportunity to test these techniques using a collection of guitar pedals and standalone effects. While this could have been done with software, we have found that using hardware is more flexible and intuitive. We also tend to find that many of the students have never seen or used hardware effects before, and that they are often surprised to find that it is possible to make music electronically without a computer.

3.4. Musicianship 1: solo performance

After introductions to the basics of sounding objects, amplification and sound effects, we begin to focus on musicianship. Most of the students are already quite accomplished musicians, so the main challenge here is to make them comfortable performing with a non-standard ‘instrument’, such as a coffee cup or toothbrush. Because many of the students have not used microphones and amplifiers in performance, we need to teach them how to handle such equipment in a concert situation. We also talk about stage presence, especially in terms of the performer who must go on stage with a non-standard instrument and sell its musical qualities to the audience.

3.5. Digital 1: Click-It

The fifth lecture is the first of three lectures on digital techniques. It starts with a brief introduction to Pure Data (PD), which is the main software platform used in the course (see section 4.1 for a discussion of software). The point here is not to teach the students everything there is to know about PD but rather to teach them just enough to be able to play with, and edit, ready-made patches. Each of the three lectures on digital techniques is devoted to one interaction type (keyboard, mouse, microphone) and one synthesis technique (delay/feedback, AM/FM, additive). The three interaction types are inspired by the ‘native’ input capacities of the laptop (Fiebrink, Wang and Cook Reference Fiebrink, Wang and Cook2007), which calls for a clutter-free starting point as opposed to using external hardware (MIDI controllers, game controllers and so on).

The first of the digital lectures focuses on the keyboard as the input device. The keyboard allows for impulsive actions and hence affords impulsive types of sounds. Inspired by the instrument/piece Clix by Ge Wang, we build an instrument called Click-It that is based on short clicks sent to a delay line with feedback. Changing the delay and feedback coefficients in turn modifies the pitch and timbre of the resultant click sounds. The instrument's mapping is designed so that typing the alphabet will increase the pitch, with the letter ‘a’ producing the lowest pitch and ‘z’ the highest. We also add an extra filter ‘blow-up’ effect when holding down the space bar. The patch is very simple, and it is easy for the students to build, yet it leads to many musically interesting sounds.

After getting the patch up and running, we expand it by adding a metronome, a counter and an array to control the amplitudes of the played clicks according to a predefined rhythmic sequence. Changing between this ‘sequence mode’ and the ‘instrument mode’ represents a good starting point for a discussion about the differences between a regular instrument, over which the musician has full control, and an interactive music device, over whose musical output it is only possible to have partial control. The instrument is programmed from scratch in class, and students themselves are encouraged to try to program along the way. However, due to the small amount of time available in class and to the fact that we want everyone to participate in performing with the instrument, there is also a ready-made version for the students to download (Figure 5).

Figure 5 Screenshot from the main patch of Click-It, a small keyboard-based instrument

3.6. Digital 2: Mousalizer

The second of the lectures on digital instruments is focused on using the mouse or trackpad as the input device. The mouse represents a device with both impulsive (button clicking) and sustained (mouse pointer) actions, and it is therefore useful for controlling both impulsive and sustained action–sound types. To keep things simple, the clicking is here used as an impulsive sound-producing action, while the sustained motion of the pointer is used for sound modification (see Jensenius, Wanderley, Godøy and Leman (Reference Jensenius, Wanderley, Godøy and Leman2010) for a summary of different types of music-related movements).

This lecture is meant to demonstrate to students that it is possible to create musically interesting sounds using only a few sine tones. By connecting the X and Y position of the mouse pointer to two separate sine tones, we can show how the two sine tones can create beat frequencies (when they approach each other) and separate tones (when they move out of the range of the critical bands).

The instrument built in this lecture is called the Mousalizer, and it is based on a simple FM synthesis combined with AM synthesis (Figure 6). In its static form, the Mousalizer is not particularly interesting sonically, but it comes to life when it is controlled with the mouse. The instrument's mapping is set up so that sound is turned on and off when one presses the mouse button, and each of the modulation parameters is controlled by the four continuous outputs from the mouse: XY position and XY change in position. The final instrument is a good example of how an otherwise limited sound engine becomes musically interesting via the addition of realtime control. Also, even though there are no coupled mappings, the use of both position and its first derivative contributes an interesting complexity to the interaction. This unpredictability appeals to students and complements the discussion of complex mappings in (Hunt, Wanderley and Paradis Reference Hunt, Wanderley and Paradis2003).

Figure 6 Screenshot of the Mousalizer instrument, based upon a simple FM/AM synthesis with real-time control from the mouse

3.7. Digital 3: Sonimotion

In the third digital lesson, we look at how we can use the built-in web camera in a laptop to control sound. Here, students learn how to generate a series of motion images based on frame differencing, and how to calculate two basic motion descriptors: quantity of motion and the XY position of the centroid of motion. We also look briefly at how these features can be used to control various synthesis parameters in real-time.

The instrument built in this lecture is called Sonimotion and is based on the ‘inverse FFT’ technique presented in (Jensenius Reference Jensenius2012). One first creates a motiongram by averaging over the rows in a series of motion images. The motiongram represents motion over time, in a fashion similar to the way in which spectrograms represent sound over time. Thus it is possible to create sound from the motiongram by treating it as if it were a spectrogram and passing each of the matrix columns of the motiongram to an oscillator bank (Figure 7). The end result is a direct and intuitive sonification of the motion. Students are also shown an interesting side-effect of this sonification technique: the possibility to use various types of video effects and filters to modify the output sound.

Figure 7 Sketch of how motiongrams can be turned into sound through an ‘inverse FFT’ technique

3.8. Musicianship 2: performing together

After having built and played with three different digital instruments, students next consider the possibilities for playing together using computers. This lecture builds on knowledge from the Oslo Laptop Orchestra, a ‘PLOrk-style’ orchestra in which each musician performs with his or her own laptop and speaker (Trueman Reference Trueman2007). We play the pieces developed in the previous lectures and explore the experiences of improvising freely, improvising with a conductor (either human or computer), and playing assignment pieces prepared by the students. The results of this lecture comprise a mid-semester public performance (Figure 8).

Figure 8 ‘Interactive Music’ students performing at a mid-semester concert at the Department of Musicology

3.9. Electronics 1: Victorian synthesiser

The last part of the course is devoted to electronics, and the first of these lectures is focused on analogue electronics in its simplest form. The lecture starts with a brief overview of electricity and electric circuits. We then play with what Collins (Reference Collins2006) calls a Victorian synthesiser, connecting two cables to a nine-volt battery and touching the two inputs on a speaker element. By adding a few more cables and various types of metal parts, we can build small circuits that in turn create rhythmic patterns. We also explore the possibilities for connecting several circuits and speaker elements together to make more complex soundscapes and long-playing rhythmic structures.

3.10. Electronics 2: Phidgets sensors

The second electronics lecture is devoted to digital electronics. After teaching with Arduino kits for some years, we now use kits from Phidgets, because the Phidgets sensor USB interfaces ship with a large collection of sensors that can be easily connected and disconnected from the board. This allows for soldering-free exploration of different types of sensors and makes it easy for students to begin working with digital electronics within a class hour.

Since there is currently no PD external available for the Phidgets kits, students use a standalone application called Phidgets2MIDI (Figure 9) to pass data to PD through MIDI. This standalone application also handles some of the basic signal processing that is necessary for working with sensor data: calculating the first and second derivatives, smoothing, removing repeating values and so on. While it would be useful for students to learn how to program all of these features themselves, we prefer to spend most of the time exploring how to control sound processes using the electronics.

Figure 9 The Phidgets2MIDI application passes data from a Phidgets interface kit as MIDI messages

3.11. Electronics 3: Acceleromagic

The third electronics lecture is devoted to showing how accelerometers can be used to sense human body motion, and how this motion, in turn, can control a digital sound engine. The goal of this lecture is to heighten the students’ awareness of different action types. The first class hour is spent on explaining the construction of an accelerometer, and how the data obtained with an accelerometer are related to position, velocity and acceleration. The second hour is spent on exploring a set of Phidgets USB accelerometers, as well as the accelerometers in the laptops and mobile phones of the students. We build an instrument called the Acceleromagic, in which the accelerometer data is used to control different types of MIDI instruments. Students learn how to extract meaningful actions from the continuous motion data stream through derivation, filtering and thresholding. This work allows them to control, for example, percussive sounds when hitting up or down, or string sounds when moving continuously from side to side.

3.12. Musicianship 3: making the show

The final course lecture is devoted to preparations for the concert. These preparations include deciding on the programme and the order of pieces as well as sorting out all of the practicalities of setting up equipment, planning rehearsals, writing programme notes and organising audio/video recordings of the show.