Hostname: page-component-745bb68f8f-l4dxg Total loading time: 0 Render date: 2025-02-06T00:47:20.118Z Has data issue: false hasContentIssue false

Towards Unified Design Guidelines for New Interfaces for Musical Expression

Published online by Cambridge University Press:  29 June 2009

Garth Paine*
Affiliation:
School of Communication Arts, University of Western Sydney, Australia
Rights & Permissions [Opens in a new window]

Abstract

The use of a laptop computer for musical performance has become widespread in the electronic music community. It brings with it many issues pertaining to the communication of musical intent. Critics argue that performances of this nature fail to engage audiences because many performers use the mouse and/or computer keyboard to control their musical works, leaving no visual cues to guide the audience as to the correlation between performance gestures and musical outcomes. The author will argue that interfaces need to communicate something of their task and that cognitive affordances (Gibson 1979) associated with the performance interface become paramount if the musical outcomes are to be perceived as clearly tied to real-time performance gestures – in other words, that the audience are witnessing the creation of the music in that moment as distinct to the manipulation of pre-recorded or pre-sequenced events. Interfaces of his kind lend themselves particularly to electroacoustic and computer music performance where timbre, texture and morphology may be paramount.

Type
Articles
Copyright
Copyright © Cambridge University Press 2009

1. Gesture

Relationships to sound are in part physical: musical instruments generally require the performer to blow, pluck, strum, squeeze, stroke, hit or bow. The acoustic instrument vibrates in a manner determined by the energy transmitted into it. The physical gestures the performer enacts generate that energy. These gestures thereby have a many faceted role: they determine the amplitude, pitch and timbre of each sound event, whilst also engaging an audience in the moment of performance. The gestures are understood to communicate an authenticity about the momentary events being created and witnessed by the audience and are often emphasised in some manner to heighten the ritualistic qualities of the communal experience that is live music performance (Davidson Reference Davidson1997; Godlovitch Reference Godlovitch1998). Such extensions of gesture are common in rock–and-roll music performances, but are not uncommon in virtuosic classical music recitals. In these cases, only a small proportion of the gesture is pragmatic and related directly to the performance of the instrument. This paper does not address the apportioning of gesture to musical control and sound generation as against flamboyant showmanship, or indeed more subtle self-expression, as this would require considerable study and experimental undertaking. It does, however, acknowledge that musicians use gesture both as a means to engage the production of sound on an instrument and as an expression of an inner intentionality. This facet of gesture is intended to convey something of the emotional interpretation the musician wishes to invoke through the nuancing of the musical material. This paper deals with instrumental control only, with a particular focus on electroacoustic performance employing innovative new interface technologies.

The author proposes that an experienced musician develops a proprioceptive relationship with his or her instrument – that is, a largely unconscious perception of movement and stimuli arising within the body from the relationship between the human body and the instrument during performance; a direct relationship is established between the physical gesture, the nature of the stimuli and the perceived outcome. The resulting awareness is multifaceted and has been at the core of musical performance for centuries. These levels of engagement extend to distributed cognition – that is, a product of the body as a whole and not simply the brain – and as such allow musicians to enjoy an embodied relationship with their instruments (where the instrument and performer may appear to dissolve into one entity), a relationship that is often communicated to the audience through performance gestures. Computer-based music, however, heralded the dislocation of the excitation, sonification mechanism, dissolving the embodied relationship the musician previously enjoyed with his or her instrument whilst simultaneously introducing a broad range of possibilities that defy the limits of the human body, raising questions about the role of gesture in musical performance and the value of haptics in successful musical instruments.

2. Interface

Playing a musical instrument causes the transfer of spatial (pitch) and temporal (duration/rhythm) information from the conscious and subconscious systems of the body to the apparatus that physically produces the sound. Any such information transfer operates from within complex traditions of culture, musical design and performance technique, and is shaped by human cognitive and motor capacities (e.g. the event speed and complex polyrhythms in the compositions of Colon NancarrowFootnote 1 (Carlsen Reference Carlsen1988; Gann Reference Gann1995; Duckworth Reference Duckworth1999), as well as personal experiences (Pressing Reference Pressing1990). Donald Norman (Reference Norman1990) refined the term affordances to refer to perceived affordances, as opposed to objective affordances. This distinction makes the concept dependent not only on the physical capabilities of the actors, but on many contemporaneous influences, including their experience and expectations, their level of attention, and perceptual ability, which in turn brings enculturation into the frame.

The mechanisation of musical instruments has a long history. Mechanical automation surfaced in the music boxes of Europe, in hand-cranked street organs, through to the theatrical extravagance of the Wurlitzer organ and the player piano. A brief overview of mechanised musical instruments would include Salomon de Caus’s pegged organ (1644); the 42 robot musicians of Johann Maelzel (holder of the patent for the metronome), for which Beethoven composed the Wellington Victory March; music boxes; and musical clocks.

Electrical automation also has a long history, dating back to the late nineteenth century with Cahill’s Telharmonium, a vast electromechanical synthesiser that occupied five train carriages when touring. Developments proceeded through various machinations to purely electronic instruments such as Dr Freidrich Adolf Trautwein’s trautonium on which Oskar Sala was a virtuoso, the ondes martenot to the theremin, made famous by the virtuosic performances of Clara Rockmore (Chadabe Reference Chadabe1997) and perhaps the most famous electronic instrument where gesture is critical in its performance. Each of these instruments retains a limited and clearly defined timbral range and a fixed morphology (Wishart Reference Wishart1996) where, even in the case of the theremin, a clear relationship between gesture and musical outcomes was evident. The performance of the theremin traditionally assigns the left hand to the control of amplitude, and the right to the control of pitch. Some timbral variation of any note can be achieved through modifying the shape of the right hand, but the synthesis engine remains unchanged and so whilst pitch is characteristically fluid on the instrument, the timbre – or, in Wishart’s sense, the morphology (the relationship between pitch, timbre and time) – remains fixed.

3. An Acoustic Instrument Model

As outlined above, prior to electronic and the digital instruments, the interface was inherently an integral and inseparable part of the instrument – part of the excitation–sonification system. For instance, all instruments in the string family share a mechanism for retaining the string which sees the string attached to the tail piece, run over the bridge, which sits on the sounding board (dispersing and amplifying the vibrations of the string), and run to the peg which, held in the scroll, allows for variations in tuning. This mechanism provides for multiple independent sound sources (four or five strings) to be held on a single instrument body. Whilst this point may seem trivial, it allows a single musician to execute multiple parallel musical ideas on a single instrument, techniques well illustrated in the solo cello suites by J. S. Bach or the solo violin works of Paganini, although in reality the design of the instrument only allows two notes to be sounded simultaneously, and a maximum of three or four notes requiring a fast arpeggiation with the notes sounded in quick succession (appearing almost simultaneous).

The development of new interfaces for electronic music performance has been impeded by the absence of a generic model for musical control of existing instruments. I see acoustic instruments as ‘successful’ interfaces for music-making, a hypothesis supported by the period of time they have persisted and the ubiquitous nature of traditional interfaces, even on new instruments (the majority of synthesisers use the traditional keyboard, electric guitars and basses are unaltered in terms of the performance interface, MIDI wind controllers use key layouts familiar to performers of clarinet, saxophone, flute and trumpet, and so on). There exists a need to combine the valuable research outcomes from the computer sciences community with the musician’s perspective at a semantic level – one approach to this problem is based on temporal data, such as the measuring of sensor pressure and angle over time, rates of velocity, acceleration and other quantifiable, measurable characteristics; however, the data itself is already conditioned by interface design decisions.

Recent work carried out by the author (Paine, Stevenson and Pearce Reference Paine, Stevenson and Pearce2007) at the University of Western Sydney sought to address these inbuilt biases by interviewing highly skilled acoustic musicians and analysing the interview data using qualitative software tools to examine the fundamental control parameters utilised by expert musicians on traditional instruments. In order to define existing models of musical gesture space, identifying direct control, levels of emergence and possible ‘uncontrol’, we sought information about how many discrete control parameters trained musicians consciously exercise in normal performance conditions. We also sought information on existing models of timbre space from the performer’s perspective by asking them how the parameters directly relate to audible timbral characteristics.

4. Participants

Semi-structured interviews were carried out with professional musicians during June 2005–July 2006. A total of 9 (n = 4 male; n = 5 female) tertiary-trained musicians participated in the initial stage of the research. All participants were involved in the teaching of their instruments, in addition to performing their instrument professionally. The length of time spent playing the instrument ranged from seven years to 30+ years. Participants were experts in the field of flute (n = 2 female); double bass (n = 4 male); violin (n = 2 female); and piano accordion (n = 1 female), and ages ranged from under 25 years to 55+ years.

A semi-structured interview schedule was devised and covered the following broad questions:

  1. (1) What instrument do you play?

  2. (2) An important aspect of learning to play your instrument is to develop control over the sound of the instrument. What aspects of the sound of your instrument are controllable?

  3. (3) When you are practising your instrument and developing your technique, what are the physical controls that you exercise in manipulating the instruments’ controllable sound properties?

  4. (4) When you are playing your instrument in a performance, what physical controls do you consciously exercise in the manipulation of the controllable sound properties?

  5. (5) To what degree are these physical controls independent or inter-related?

Participants were recruited through professional music organisations, instrument and teacher associations and professional orchestras. Those that were interested were informed about the study, its goals and the interview process. Each participant was issued with an informed consent form, a participant information sheet and a general demographic questionnaire. Interview times ranged from 35 to 110 minutes in length.

5. Stage One Results

The transcripts of these interviews were analysed for musical concepts using Leximancer (Smith Reference Smith2007), a qualitative analysis software solution suited to emergent methodologies for undertaking discourse analysis, grounded theory, action research, conversation analysis, ethnography, phenomenology and mixed-methods research.

A basic Leximancer analysis of the initial interviews (September 2005) identified a list of shared terms used by musicians to define the principal musical parameters of the target acoustic instruments as:

  • tone (tone colour, sound colour (resonance), tone quality)

  • dynamics

  • volume

  • expression

  • duration

  • vibrato

  • articulation

  • attack

  • release

  • sustain

  • pitch and

  • intonation.

The concepts are based on a lexicon of seed words entered by the researcher(s) and the software’s concept learning routine, which discovers relevant text segments that do not contain the actual seed terms identified by the user, providing automatic taxonomy discovery and concept cluster mapping by applying Bayesian theory to the interview transcripts.

Automated concept mapping was undertaken to discover primary control mechanisms on successful acoustic instruments. The Leximancer concept map can be adjusted to show differing levels of concordance, subsequently varying the number of concepts displayed. In this way the map in figure 1 can be reduced to that shown in figure 2, which is useful in determining the primary concepts and developing hypotheses from the interview data.

Figure 1 Leximancer concept map.

Figure 2 Leximancer concept map reduced to principal control considerations for the flute.

It can be seen in figure 1 that concept clustering places the two flute players on one side of the concept map and the violin and double bass on the other. Equally, concepts specific to these instruments (air for the flute and bow for the strings) are weighted towards the appropriate instruments, whilst shared concepts such as timbre, sound, pitch, control, tone, pressure and colour indicate concordances for all interviewees. The concept maps were continually refined in an effort to gain greater clarity regarding the relationships between control parameters and audible timbral characteristics; however, it can be seen that relationships have been established using asymmetric concordance and that these concepts appear to remain musically useful.

This study was undertaken to assist in developing a mapping strategy for the ThummerFootnote 2 interface. The study was called the Thummer Mapping Project (ThuMP) and facilitated an exploration of the playability of high-dimensional control spaces from a performer’s perspective by mapping control dimensions to timbral variables that were cognitively meaningful to those musicians. The ‘Playability’ studies outlined in Paine, Stevenson and Pearce Reference Paine, Stevenson and Pearce2007 fed iteratively back into the mapping strategies, providing a performer perspective, determining the feasibility and value of control mappings on the basis of musical literacy.

6. The ThuMP Model

Due to the author’s concerns that the stage one results had been heavily conditioned by his own conservatorium training, a second analysis was undertaken by a skilled qualitative analyst with no musical training, using NvivoFootnote 3 software. The stage two analysis led to the models presented in figure 3, figure 4 and figure 5. These models of instrument control are broken into two stages:

Figure 3 Musical parameter overview.

Figure 4 Control parameters model.

Figure 5 Detail of the control parameters for dynamics.

  1. (1) musical parameters; dynamics, pitch, vibrato, articulation and attack/release being identified as the focus of the physical instrument control. A good command of these parameters was also identified as key in achieving a well-developed instrumental tone, the principle concern for all interviewed musicians; and

  2. (2) an identification of four predominant physical parameters that were commonly aggregated to bring about the musical attributes listed above. The physical parameters were pressure, speed, angle and position.

The model represented in figure 3 also indicates the role musical expression or musicianship plays in aesthetic decisions, and the manner in which these notions act as an overall metric for the musical sensibility that underlies all western instrumental training and musical decision-making.

The above musical parameters were analysed in terms of the physical control musicians employ in order to achieve musical outcomes. For instance, one subject commented that ‘pitch is controlled by the position of the bow, as well as the movement of the bow between the bridge and the fingerboard’, a flute player commented that a ‘fast air stream produces a dark tone, whilst a relaxed, slow air stream produces a lighter, softer tone. The higher pitch registers are achieved by using a faster air stream, angled slightly upwards, whilst the lower registers require a slower downward air stream’. An analysis of such statements led to the identification of four primary physical controls; pressure, speed, angle and position (figure 4 and figure 5).

Many of these instrumental techniques could also be identified in the instrumental technique literature (principle, for instance, within the flute literature would be Kincaid and Polin Reference Kincaid and Polin1967; Chapman Reference Chapman1973; Quantz and Reilly Reference Quantz and Reilly1975 and Morris Reference Morris1991). Relying on such literature, however, takes the underlying model as a given – that is, accepts it as the predominant practice. This is not necessarily so, as technique no doubt evolves. It is for this reason that we sought interviews with professional musicians and derived the model presented here from that material alone. The literature on instrumental technique was not ignored, simply used as a secondary source.

7. Applying the Model in Computer Music

The excitation–sonification relationship is broken into interface and synthesis algorithm in computer music (Roads Reference Roads1996; Chadabe Reference Chadabe1997). Jeff Pressing’s article ‘Cybernetic Issues in Interactive Performance Systems’ (Pressing Reference Pressing1990) outlined a model for an interactive performance system that defined a musical instrument within such a system (figure 6). His model contains a control surface, a processor and an effector mechanism. He comments that:

The parts of the instrument that are directly controlled or manipulated by parts of the body, and to which information is directly transferred, are called the control interface. The parts that actually produced the sound are called the effector mechanism. Intervening between the control interface and effector mechanisms is often a processor of some kind that converts information in control format to effector format (yielding appropriate form, range and sensitivity).

Traditional instruments have a nearly one-to-one response between actions of the performer and the resulting sound, a stimulus–response model fits well. Interaction between the person and the instrument takes place through the aural [visual] feedback loop and the performer makes decisions on that basis in real-time. (Pressing Reference Pressing1990: 12)

Mulder (Reference Mulder1989) expands the Pressing model in several important ways (figure 7):

Figure 6 Pressing model for performer–instrument system.

  1. (1) he includes the audience within the model;

  2. (2) he breaks the interaction between performer, instrument and audience down in such a way as to reflect intention and reception; and

  3. (3) he defines the instrument as a collection of sensors, actuators, processing and sound generation, a much more explicit division of the elements that combine under the nomenclature of instrument, performer and audience.

Extending the consideration of performance interface models, the direct control of a large number of synthesis parameters is an impossible and not necessarily musical task without some form of correlation into higher-order control patterns. One approach that continues to increase in popularity is to develop preset interpolation interfaces such as those available in the Kyma software (Scaletti Reference Scaletti2004) and in AudioMulch (Bencina Reference Bencina2003), and audio plugins such as GRM Tools (INA-GRM and Favreau Reference Favreau2008) and the INT.LIB for Max/MSP (Larkin Reference Larkin2007).

Figure 7 Mulder model of interactive electronic musical interface.

Oliver Larkin comments that the design goals for INT.LIB were as follows:

  • to allow the control of multiple parameter sets independently from one encapsulated interface;

  • to abstract the user interface from the max patch;

  • to facilitate rapid layout of the interpolation space;

  • to be fast enough to support interpolation of many presets featuring many parameters; and

  • to be easy to understand and use. (Larkin Reference Larkin2007)

Ross Bencina outlines the Metasurface in AudioMulch, addressing similar concerns to Larkin:

The Metasurface – a mapping interface supporting interactive design of two-to-many mappings through the placement and interpolation of parameter snapshots on a plane. The Metasurface employs natural neighbor interpolation, a local interpolation method based on Voronoi tessellation, to interpolate between parameter snapshots. (Bencina Reference Bencina2005: 101)

In line with Pressing and Mulder, the selection of an interface that provides physical control parameters as defined in the ThuMP study (pressure, speed, angle and position), combined with the careful mapping of the parameter space (both direct and through preset interpolation as per Scaletti Reference Scaletti2004; Bencina Reference Bencina2005 and Larkin Reference Larkin2007), may encourage a link between generation (gestural quality – intense, fast, light, etc.) and musical outcome.

In order to synthesise the knowledge the models provide into an understanding of the issues pertaining to real-time performance, the author has been experimenting with the application of the above physical control parameter model for electronic music performance using experimental interfaces such as the Intuos3 Wacom Graphics TabletFootnote 4 and the Nintendo Wii RemoteFootnote 5 (WiiMote). This discussion will focus on the WiiMote.

8. Nintendo Wii Remote

The Nintendo Wii Remote (WiiMote) has been the focus of an explosion of music performance, DJ and VJ experimentation. Software frameworks quickly appeared for the Macintosh computerFootnote 6 and the WindowsFootnote 7 and LinuxFootnote 8 operating systems and were compiled into objects or plugins for programming languages such as Max/MSP, Quartz Composer (QCWii), Isadora, OSCulator (providing an OSCFootnote 9 bridge), and embedded in applications such as WiiToMIDI, Wiinstrument and Wii Loop Machine (actually developed in Max/MSP).

Part of the attraction is that the interface is wireless (Bluetooth), but a major factor will also have been the range of control afforded by the WiiMote. The WiiMote contains eleven buttons (momentary) and a three-dimensional accelerometer (six continuous streams of data, three direct and three second-order parameters (pitch, roll yaw – see Table 1). In addition to this already substantial data set, the WiiMote accepts an accessory called a nunchuck, which is held in the remaining free hand and contains a second three-dimensional accelerometer, a traditional two-dimensional joystick and two trigger-style buttons (C and Z).

Table 1 WiiMote and nunchuck control parameters.

One of the characteristics of such an interface is that the buttons and or the joystick can be used in parallel with, and independent of, the three-dimensional accelerometer. This means that the data streams shown in Table 1 can be performed semi-independently and simultaneously.

An infrared Sensor Bar (WiiBar) can be added to this setup to sense the absolute position of the WiiMote on the X and Y axes, providing an additional two continuous data elements. Table 1 indicates that the WiiMote can produce up to nine simultaneous but partly inter-related data streams (6 continuous and 2 or 3 momentary; it may be possible to actuate more than 2 or 3 buttons simultaneously but this is a constraint of finger flexibility rather than the interface transmission scheme) and the nunchuck can produce up to ten simultaneous but partly inter-related data streams (8 continuous and 2 momentary). When combined with the WiiBar a data set of 16 continuous and 5 momentary elements can be produced simultaneously.

Table 2 Analysis and categorisation of WiiMote data.

When considered as un-correlated control, this represents too many pieces of data for an individual to use constructively at any one time. In order to consider the WiiMote–nunchuck combination in terms of the acoustic instrument model outlined above, an analysis of the parameter space was undertaken in terms of pressure, speed, angle and position (Table 2).

As the continuous data streams are simultaneous but partly inter-related, some decisions need to be made regarding the use of control parameters in synthesis where by their musical outcomes could conflict. For instance the XAccel/YAccel/ZAccel data streams cannot be independent – a movement in any one axis will cause some variation in the other axis. Pitch, Roll and Yaw may also be affected by accelerative movements in the X/Y/Z planes; however, within themselves they present more independence that the acceleration data, which in all planes is subject to gravity. The buttons are of course independent although only momentary. The joystick data, XJoy, Yjoy, is the only truly independent continuous data stream available. Of course considerable skill is required to maintain total independence over X- and Y-axis movement using the joystick.

It should also be noted at this point that the more advanced Wacom tablets such as the Intuous3 provide for at least twelve pieces of continuous data and two buttons as is illustrated in the Wacom pen assignment page in the Kyma software (see figure 8). The author uses a Wacom tablet as his principle interface for electroacoustic performance, but anecdotal evidence suggests that audiences still find relationship of the gesture to musical outcome problematic, failing to resolve issues pertaining to authenticity of performance.

Figure 8 Kyma software Wacom pen status screen.

This issue was reinforced by a recent review of concert for the ensemble SynC, in which the author performs using both a Wacom Tablet and a WiiMote/nunchuck combination.

This emphasis on performativity was a key feature of Paine’s work in particular. Use of the spatially-operated Wii controller obviously lends a degree of theatricality to a work – in Paine’s case, this was characterised by a precise, restrained motility that had a more obvious connection to the sounds produced than is necessarily the case with other spatial instruments such as the theramin. As far as visual musicality goes, the Wii has an edge over the incongruent sight of a performer grooving along to the minute scratching motions of a stylus on a drawing tablet. (Davis Reference Davis2008)

The apparent ‘connection to the sounds produced’ is seen here to be of importance to the reviewer who, being more attuned to classical musicology than experimental music performance, serves as an excellent guide to the predominant expectations of audiences.

9. Instrument Design

Once a controller has been selected and the controller parameters are understood – a process of exploring the inter-relatedness of the parameters, the temporal qualities of the parameters streams and the kinds of inter-parameter complexity available – the instrument design begins. In addition to the above discussion, electronic music performance differs from acoustic in that the composer designs and builds the instruments to suit the demands of the composition. The gestural quality of interface parameter sets plays an important part in the associated gesture of the musical outcomes – slow or fast transitions, fast chaotic timbral changes, accuracy in pitch or amplitude and repeatability are just some of the considerations at hand.

Instrument design requires some serious consideration of musical aesthetic and the relationship between performer and audience. The interface and its implementation therefore serve two primary goals:

  1. (1) to increase performability, allowing the musician to nuance musical outcomes in a way not possible with existing interfaces or using the mouse–keyboard computer interface; and

  2. (2) to increase communication with the audience, displaying something of the energy and intent of the performer, providing a conduit for engagement in the real-time qualities of the performance: in other words, the ritual of performance (Godlovitch Reference Godlovitch1998; Borgo Reference Borgo2005).

Any implementation of a new musical interface must therefore consider the ecology of this environment. Gurevich and Treviño (Reference Gurevich and Treviño2007) discussed the development of a framework for an ecology of musical action:

An ecological framework without the assumption of a commodity or a singular creator makes it admittedly difficult to unify or relate the experiences of the individual actors in the system. Donald Norman’s (Norman Reference Norman2004) formulation of three levels of processing in the human brain and associated modes of experience facilitates a meaningfully descriptive but inclusive consideration of the musical experience from a variety of points of view. The three levels of processing are visceral, automatic and pre-wired reactions to sensory stimuli; behavioral, involved in the subconscious control of learned everyday actions (driving a car, taping, playing a violin); and reflective, the highest-level conscious thought in which we form opinions, plans, and abstractions. Organized in a hierarchy, adjacent levels can inform one another, but control acts downward. The reflective level tries to influence behavior based on conscious thought, and the behavioral level can in turn try to ‘enhance and inhibit’ the visceral. While Norman argues that good design requires a balanced appeal on all three levels, it is also clear that all three levels are engaged in creating music. … Norman describes the skilled performer’s ability to play a piece unconsciously (behavioral) while simultaneously considering matters of the large-scale form (reflective). The listener reacts viscerally to the sound and may also contemplate meaning. (Gurevich and Treviño Reference Gurevich and Treviño2007: 109)

Gurevich and Treviño point to the difficulty of identifying a unified source for musical creation in a complex system where the interface and the sonification mechanism are separated. Their adaption of the three levels of processing outlined by Norman (Reference Norman2004) illustrate some of the cognitive associations brought into play when engaging both as a performer and an audience member during a musical performance.

Visceral and behavioral levels are enshrined in the kinetic gesturing that brings about the musical outcomes, representing a sonification of the performative gesture. The reflective layer is brought to bear by the musician who is actively but subconsciously planning form, structure and harmonic progression. The momentary and the future abstract are always coexistent and active.

The author supports Gurevich and Treviño’s statement that musicians are working at all three levels identified by Norman when improvising and performing dynamic scores, and was encouraged by the comprehensiveness of their approach. Whilst it is not usually possible to consider all three levels within a single experimental design, it is critical that any study is contextualised within cultural mores.

Gurevich and Treviño also contextualise Norman’s three levels of processing in terms of interface evaluation:

Norman’s three levels of processing offer a new currency for describing the experience of music creation that places the electronic music interface appropriately in context. This framework has three distinct advantages: 1) it admits a broader range of aesthetic concerns; 2) it provides a more meaningful way to ‘evaluate’ an interface; and 3) it expands the scope for the consideration of novel interfaces. (Gurevich and Treviño Reference Gurevich and Treviño2007: 109)

In addition to the visceral and affective considerations, Schwartz and Godfrey (Reference Schwartz and Godfrey1993) define seven principle concepts in contemporary composition which come from the acoustic instrumental paradigm, but if one is to bridge the experimental–classical divide, and/or to develop an interface or instrument that may have broad application, it needs to be possible to execute these musical notions: pitch logic, time, sound colour, texture, process, performance ritual and parody (historicism).

In developing an approach to the WiiMote, the author considered these musical concepts in terms of both instrument design and mapping strategies for musical interfaces.

10. Where is the Instrument?

Oritz Pérez, Knapp and Alcorn discuss this issue when developing a composition-driven approach to using novel interfaces in Díamair for choir and integral music controller (Ortiz Pérez, Knapp and Alcorn Reference Ortiz Pérez, Knapp and Alcorn2007). The diagrams in figure 9 illustrate the way in which the compositional decisions influence the interface and in turn how the interface design influences the instrument design (software synthesis).

Figure 9 Composition-driven approachs to using novel interfaces (Ortiz Pérez et al. 2007).

A combination of these considerations is brought to bear when developing a musical interface that also addresses compositional constraints. One example is the author’s work with Michael Atherton in their ensemble SynC.Footnote 10 The composition titled Encounter, for hurdy-gurdy and live electronic processing from the Parallel Lines CD (Paine and Atherton Reference Paine and Atherton2006), utilised the Capybara/Kyma systemFootnote 11 for live electronic processing of the hurdy-gurdy sound, and implemented the Nintendo WiiMote as the control interface.

11. Encounter – an example of the WiiMote as musical controller

In developing the synthesis mapping for the WiiMote in this musical work, a number of compositional strategies was considered. The composition (or rather the synthesis algorithms it contains) comprise a large number of synthesis variables. It was not possible to constrain the variables in such a manner that they could all be controlled in real-time without grouping variables into relationships that, whilst appropriate to one section of the composition, imposed inter-relationships that were musically inappropriate in other sections. To address this issue, the author chose to automate some variables by making preset snapshots containing those variables whilst leaving others under real-time control. The parameters under real-time control were: filter centre frequency, the control of buffer recordings (four separate buffers), the rate and density of granulation, the playback of a single sample (one pre-recorded sample – rattling of bamboo – is used in the work), the sample-and-hold rate of two oscillators and the delay feedback time associated with these two oscillator instruments, and the control of two frequency/pitch variables of those oscillators which form the solo electronic instrument in the central section of the work. A few timbral variables, including granulation buffer freeze (providing control over the FFT re-synthesis rate, producing timbral stretch), a brassage effect, controlled by the ChopA variable were also allocated during the work. See figures 10 and 11.

Figure 10 OSCulator and WiiMote mappings to Kyma/Capybara variables for the musical work Encounter.

Two buttons were also allocated to step through preset states determining the setting of the variables not under real-time control.

Figure 11 A vocoder object within the Kyma patch for Encounter showing the use of some of the defined real-time variable control.

The interface for this work is shown in figure 12, where the buttons indicating the state of the record buffers occupy the top row. Variables controlled in real-time directly by the WiiMote are indicated either as a button or outlined with a black box, as can be seen in the row of potentiometers at the very top of the window, one of which, WispCross, is controlled by the Nun.Joy.Y, or the other variables below such as RCFreqMod. It should also be noted that most of the Waituntil functions used in the work are noted in the top right of the GUI, and can be seen in the top line of the timeline in figure 13. These functions stop the progression of the timeline, whilst keeping all instantiated algorithms live; in essence, they form a dynamic scheduling framework for non-determinate and non-linear temporal structuring of musical material in performance.

Figure 12 Kyma interface for Encounter.

Figure 13 Kyma timeline for Encounter – all tracks contain real-time algorithms, not pre-recorded sound.

This article is not the place to discuss all the creative decisions the composer made in developing Encounter for hurdy-gurdy and live electronics. However, the composition contains a number of options for re-mediating the hurdy-gurdy sound, which include the possibility of capturing performed phrases for replay (establishing a dialogue between the digital and acoustic proponents) and sculpting variations upon those phrases that reflect the timbral, viscous potential of computer-based digital re-mediation of the acoustic sound. The choice to have both event-driven, pointillistic (staccato, legato) pitch/noise response options as well as textural, timbral alternatives was driven by a desire to have a wide range of musical possibilities at hand during performance. The composition was conceived as a comprovisation (Morris Reference Morris1987; Priest Reference Priest2005; Atherton Reference Atherton2006), drawing on influences from aleatoric music, through Fluxus, notably Earle Brown, John Cage and Terry Riley (in C), whereby aesthetic decisions pertaining to timbral space are determined in the composition process but the navigation of those potentials into a temporal form occurs through structured improvisation. These composed potentials were further defined into three sections, a timbral hurdy-gurdy opening, growing from the acoustic sound to a larger diffused and augmented sound field (Emmerson Reference Emmerson1996), followed by an event-driven section in which an oscillator-based electronic instrument predominates, leading to a breaking down of the pitch space into more noise-based sounds resolving into the final section of the work, which draws primary characteristics from the first timbral section. The composition uses non-linear control of temporal events, so these sections do not have to proceed in this manner in performance, but the wide range of potentials inherent therein have been established through the composition process.

12. Conclusion

This paper focused on issues pertaining to the development of performance interfaces that provide sufficiently convincing gestural control affordances to overcome any concern about authenticity in performance whilst providing the potential for highly nuanced, expressive, embodied music performances.

The discussion of the ThuMP project presented a brief outline of a new approach to these issues, introducing a model for musical control developed from a musician’s perspective. This model encourages the design of computer music performance interfaces that utilise a gestural language for controlling/creating live electronic music on a laptop computer derived from a musical rather than an engineering or computer science perspective whilst addressing the many contemporaneous influences noted in the discussion about affordances (Gibson Reference Gibson1979; Norman Reference Norman1990).

The identified inter-relationship of all musical parameters exemplifies a dynamical system. The relationship between the complexity of control parameters and the evolving nature of musical practice has also been discussed with specific reference to the notion of dynamic morphology (Wishart Reference Wishart1996), addressing both musical material and the notion of morphology of gesture in musical control, and non-linear approaches to musical organisation as composition and performance (Paine Reference Paine2002, Reference Paine2004, Reference Paine2006, Reference Paine2007b).

The adoption of interface technologies such as the WiiMote and the Wacom graphics tablet for laptop music performance makes the consideration of morphological approaches to musical interfaces an imperative (Paine Reference Paine2007a). The extraordinarily swift adoption of these interfaces for laptop music performance is a clear indication that gestural control is seen as important to both musicians and audiences alike, and remains one of the most intricate and complex areas of development in laptop music performance tools.

The common physical instrument controls identified in the ThuMP – pressure, speed, angle and position – were discussed. Not only do they represent the key variables in controlling the timbre of the instrument (i.e. dynamics, pitch, vibrato, articulation (including attack and release)), but they also correspond with key cognitive affordances (Gibson Reference Gibson1979; Norman Reference Norman1990) associated with playability and control mapping; affordances that have developed over several centuries in instruments that have persevered and that provide discernable, just-noticeable-difference control over timbre, pitch, amplitude and articulation, both individually and in combination.

In the light of these analyses, the author examined the way in which pressure, speed, angle and position could act as design guidelines for future interface development and the application of the WiiMote and the Intuos3 Wacom Tablet as musical interfaces.

The musical work Encounter was discussed as a means of illustrating the application of the proposed models to a concrete example, synthesising the knowledge the models provide into an understanding of the issues pertaining to real-time performance.

Acknowledgements

I would like to thank the MARCS Auditory labs and colleagues Ian Stevenson from the School of Communication arts and Angela Pearce from the School of Psychology at the University of Western Sydney for their assistance during the ThuMP project.

Footnotes

1 See http://en.wikipedia.org/wiki/Conlon_Nancarrow (accessed 20 October 2008).

2 See http://www.thummer.com (accessed 20 October 2008).

3 See http://www.qsrinternational.com (accessed 20 October 2008).

4 See http://www.wacom.com/intuos (accessed 20 October 2008).

5 See http://www.nintendo.com/wii/what/accessories (accessed 20 October 2008).

6 See http://www.wiili.org/index.php/DarwinRemote (accessed 13 September 2007).

7 See GlovePIE and RMX Automation.

8 See WMD and Cwiid.

9 Open Sound Control (OSC) is a network protocol common in music applications. See http://www.osc.com (accessed 13 September 2007).

10 See http://www.syncsonics.com (accessed 20 October 2008).

11 See http://www.symbolicsound.com (accessed 20 October 2008).

References

Atherton, M. 2006. At the Edge of what Comes Next: ‘Comprovised’ Meaningfulness in Jiriyai, a New Work for Percussionist and Dancer. In B. Crossman, S. Macarthur and R. Morelos (eds.) Intercultural Music: Creation and Interpretation. Sydney: Australian Music Centre, 8389.Google Scholar
Bencina, R. 2003. Audiomulch: Interactive Music Studio. obtained online: http://www.audiomulch.com (accessed 26 June 2008).Google Scholar
Bencina, R. 2005. The Metasurface – Applying Natural Neighbour Interpolation to TwotoMany. Paper presented at the International Conference on New Interfaces for Musical Expression (NIME-05), Vancouver, BC, Canada.Google Scholar
Borgo, D. 2005. Sync or Swarm: Improvising Music In A Complex Age. New York: Continuum.Google Scholar
Carlsen, P. 1988. The Player-piano Music of Conlon Nancarrow: An Analysis of Selected Studies. ISAM monographs, no. 26. Brooklyn, NY: Institute for Studies in American Music, Conservatory of Music, Brooklyn College of the City University of New York.Google Scholar
Chadabe, J. 1997. Electric Sound: The Past and Promise of Electronic Music. New Jersey: Prentice Hall.Google Scholar
Chapman, F. B. 1973. Flute Technique (4th ed.). London, New York: Oxford University Press.Google Scholar
Davidson, J. W. 1997. The Social in Musical Performance. In D. Hargreaves and A. North (eds.) The Social Psychology of Music. New York: Oxford University Presspp. 209228.CrossRefGoogle Scholar
Davis, A. 2008. Everything but the Kitchen SynC… . Obtained online: http://www.resonatemagazine.com.au/article/everything-but-the-kitchen-syn-c.html (accessed 21 October 2008).Google Scholar
Duckworth, W. 1999. Talking Music: Conversations with John Cage, Philip Glass, Laurie Anderson, and Five Generations of American Experimental Composers. New York: Da Capo Press.Google Scholar
Emmerson, S. 1996. Local/Field: Towards A Typology Of Live Electronic Music. Journal of Electroacoustic Music 9: 1012.Google Scholar
Gann, K. 1995. Music in the Twentieth Century series. The Music of Conlon Nancarrow. Cambridge, New York: Cambridge University Press.Google Scholar
Gibson, J. J. 1979. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin.Google Scholar
Godlovitch, S. 1998. Musical Performance: A Philosophical Study. London, New York: Routledge.Google Scholar
Gurevich, M. Treviño, J. 2007. Expression and Its Discontents: Toward an Ecology of Musical Creation. Paper presented at the International Conference on New Interfaces for Musical Expression (NIME-07), New York.CrossRefGoogle Scholar
INA-GRM and Favreau, F. 2008. GRM Tools. Obtained online: http://www.grmtools.org/index.html (accessed 21 October 2008).Google Scholar
Kincaid, W. Polin, C. C. J. 1967. The Art and Practice of Modern Flute Technique. New York: MCA Music.Google Scholar
Larkin, O. 2007. INT.LIB – A Graphical Preset Interpolator For Max Msp. Paper presented at the International Computer Music Conference, Copenhagen.Google Scholar
Morris, B. 1987. The New York Comprovisation. Obtained online: http://www.archive.org/details/MC_ButchMorris (accessed 21 October 2008).Google Scholar
Morris, G. 1991. Flute Technique. Oxford, New York: Oxford University Press.CrossRefGoogle Scholar
Mulder, A. 1989. Design of Virtual Three-dimensional Instruments for Sound Control. Unpublished PhD, Simon Fraser University, Burnaby, BC, Canada.Google Scholar
Norman, D. A. 1990. The Design of Everyday Things. New York: Doubleday.Google Scholar
Norman, D. A. 2004. Emotional Design. New York: Basic Books.Google Scholar
Ortiz Pérez, M. A., Knapp, B. Alcorn, M. 2007. Díamair: Composing for Choir and Integral Music Controller. Paper presented at the International Conference on New Interfaces for Musical Expression (NIME-07), New York.CrossRefGoogle Scholar
Paine, G. 2002. Interactivity, Where To from Here? Organised Sound 7(3): 295304.CrossRefGoogle Scholar
Paine, G. 2004. Gesture and Musical Interaction: Interactive Engagement through Dynamic Morphology. Paper presented at the New Interfaces for Musical Expression, 2004, Hamamatsu.Google Scholar
Paine, G. 2006. Interactive, Responsive Environments: A Broader Artistic Context. In Engineering Nature: Art & Consciousness in the Post-biological Era. Chicago: University of Chicago Press, Intellect LTD, 312334.Google Scholar
Paine, G. 2007a. Interfacing for Dynamic Morphology in Computer Music Performance. Paper presented at the International Conference on Music Communication Science (ICOMCS), Sydney.Google Scholar
Paine, G. 2007b. Sonic Immersion: Interactive Engagement in Real-Time Immersive Environments. SCAN Journal of Media Arts and Culture 4 (1) http://scan.net.au/scan/journal/display.php?journal_id=90 (accessed 20 April 2009).Google Scholar
Paine, G. Atherton, M. 2006. Parallel Lines (musical composition). Tucson, Arizona: Celestial Harmonies.Google Scholar
Paine, G., Stevenson, I. Pearce, A. 2007. The Thummer Mapping Project (ThuMP). Paper presented at the International Conference on New Interfaces for Musical Expression (NIME-07), New York City.CrossRefGoogle Scholar
Pressing, J. 1990. Cybernetic Issues in Interactive Performance Systems. Computer Music Journal 14(1): 1225.CrossRefGoogle Scholar
Priest, G. 2005. Composing Potentials: Gail Priest talks with Garth Paine. Realtime 67: 12.Google Scholar
Quantz, J. J. Reilly, E. R. 1975. On Playing the Flute. New York: Schirmer Books.Google Scholar
Roads, C. (ed.) 1996. The Computer Music Tutorial. Cambridge, MA: The MIT Press.Google Scholar
Scaletti, C. 2004. Kyma X Revealed: Secrets of the Kyma Sound Design Language. Champaign, IL: Symbolic Sound.Google Scholar
Schwartz, E. Godfrey, D. 1993. Music since 1945: Issues, Materials, and Literature. New York: Schirmer, Thomson Learning.Google Scholar
Smith, D. A. 2007. Leximancer. Obtained online: http://www.leximancer.com/cms (accessed 21 October 2008).Google Scholar
Wishart, T. 1996. On Sonic Art. Philadelphia, PA: Harwood.CrossRefGoogle Scholar
Figure 0

Figure 1 Leximancer concept map.

Figure 1

Figure 2 Leximancer concept map reduced to principal control considerations for the flute.

Figure 2

Figure 3 Musical parameter overview.

Figure 3

Figure 4 Control parameters model.

Figure 4

Figure 5 Detail of the control parameters for dynamics.

Figure 5

Figure 6 Pressing model for performer–instrument system.

Figure 6

Figure 7 Mulder model of interactive electronic musical interface.

Figure 7

Table 1 WiiMote and nunchuck control parameters.

Figure 8

Table 2 Analysis and categorisation of WiiMote data.

Figure 9

Figure 8 Kyma software Wacom pen status screen.

Figure 10

Figure 9 Composition-driven approachs to using novel interfaces (Ortiz Pérez et al. 2007).

Figure 11

Figure 10 OSCulator and WiiMote mappings to Kyma/Capybara variables for the musical work Encounter.

Figure 12

Figure 11 A vocoder object within the Kyma patch for Encounter showing the use of some of the defined real-time variable control.

Figure 13

Figure 12 Kyma interface for Encounter.

Figure 14

Figure 13 Kyma timeline for Encounter – all tracks contain real-time algorithms, not pre-recorded sound.