2. Embodied Cognition

The embodied cognition hypothesis that has become important in cognitive science represents a number of similar and inter-related approaches to cognition (Varela, Thompson and Rosch Reference Varela, Thompson and Rosch1991; Hampe and Grady Reference Hampe and Grady2005), which derive from a shared belief in ‘experientialism’. Experientialism is the theory that knowledge derives from first-hand experience. The embodied hypothesis generally holds that the mind is thoroughly defined by the human body and perceptual faculties; bodily experiences provide the language by which cognitive processing unfolds. Cognitivism is the more traditionally popular alternative to embodied cognition. It derives from positivism: the view that knowledge drives from logical and mathematical analysis of experience. Cognitivism views the mind and body as fundamentally opposed, and thought as the computation of arbitrary symbols (Gardner Reference Gardner1987).

3. Historical roots of disembodiment in music technology

Cognitivism is deeply entrenched in Western cultural history (Todes Reference Todes2001; Uttal Reference Uttal2004), and has become the dominant paradigm in science and the arts (Damasio Reference Damasio2008; Leman Reference Leman2008; McGilchrist Reference McGilchrist2009). The influence of cognitivism has shaped phenomenology, psychology and cognitive science (Gardner Reference Gardner1987; Todes Reference Todes2001). Though beyond the scope of this paper, a detailed account of these points as they relate to music and sonification is offered by Worrall (Reference Worrall2010).

Descartes made a distinction between the mathematical and physical aspects of music in 1618 (Descartes Reference Descartes1647). He detached these from their subjective counterparts, which he thought were unworthy of scientific analysis (Augst Reference Augst1965). His philosophy had a profound influence on Immanuel Kant's aesthetic theory. Kant believed in a universal formal aesthetics and logic that are independent of human cognition (Kant Reference Kant1929). Schopenhauer considered music to be a disembodied manifestation of ‘the will’, a ‘noumenal’ or objective reality represented to cognition by phenomena. This served to re-enforce subject–object duality in musical thinking (Schopenhauer 1844; Magee Reference Magee1999). Richard Wagner subscribed to Schopenhauer's view of music as the manifestation of an objective reality (Darcy Reference Darcy1994). Arnold Schoenberg would borrow from Wagner's musical philosophy and expand his Hauptmotife technique (Brand and Hailey Reference Brand and Hailey1997) into serialism. Both of these techniques focus on the objective, positivistic formalisation of musical structure in accordance with some assumed universal aesthetic code. Anton Webern and Pierre Boulez inherited and further extended Schoenberg's techniques and thinking in pursuit of an idealistically ‘democratic’ serial music (Grant Reference Grant2005). Through these developments, disembodied and positivistic models came to dominate the landscape of Western art music, setting a dualistic tone for further musical and technological developments to come during the mid-1900s.

Pierre Schaeffer's adapted Husserl's epoché (Reference Husserl1931) in his ‘reduced listening’: a mode of listening that requires the suspension of judgement on the source of a sound in order to reveal more about the sound itself. Such sounds, theoretically decoupled from their sources, he termed objets sonores (sound objects). This concept carries with it dualistic assumptions. Steeped in Husserl's disembodied worldview, it asserts a division between subjective sound and objective cause (Kane Reference Kane2007). A new field of acousmatic (‘behind the veil’) music grew from the application of this methodology.

Early proponents of musique concrète recorded sound objects to tape for use as materials in a compositional process determined by the reduced listening method. In their decoupled context, sound objects were explored and elaborated through tape manipulation techniques like cutting and pasting, looping, reverb, speed manipulation and eventually overdubbing. Many groundbreaking techniques pioneered by Schaeffer and others during the 1950s were built around the same disembodied assumptions as reduced listening and epoché. Today disembodiment in music technology is a global phenomenon (Terrugi Reference Terrugi2007). Data sonification is often undertaken using these disembodied techniques and technologies. The mapping problem is a result of the positivistic cognitivism which auditory display has inherited from these technologies.

4. Aesthetics in Auditory Display

Aesthetic issues were less explored in the early days of sonification research. Today, they are gaining an appreciation within the field (Vickers and Hogg Reference Vickers and Hogg2006). However, despite some notable examples to the contrary (Goßman Reference Goßman2010; Barrass and Vickers Reference Barrass and Vickers2011), the discussion on aesthetics rarely ranges beyond the classification of individual sonifications as ‘aesthetically pleasing’. Aesthetics are treated as a means of reducing annoyance and guaranteeing listener engagement. The general rule of thumb is to ‘design aesthetically pleasing (e.g., musical, etc.) sonifications to the extent possible while still conveying the intended message’ (Walker and Nees Reference Walker and Nees2011). Such approaches emphasise the cosmetic value of aesthetics without fully exploring the potential they offer for more semantically rich and complex auditory displays. Aesthetic concerns no doubt weigh heavily on the design of attractive sonifications, but an important finding by Leplâtre and McGregor (Reference Leplâtre and McGregor2004) shows that aesthetics and functionality cannot be dealt with independently in auditory display. This highlights aesthetics as being more important to the sonificaiton process itself than was previously accepted.

Music and sound art practitioners have harnessed sonification (and sonification-like processes) as an artistic technique (see Xenakis Reference Xenakis1985; Dunn and Clark Reference Dunn and Clark1999; Quinn Reference Quinn2001; McKinnon Reference McKinnon2013). Sonification also turns to these arts to inform its aesthetic choices (Childs Reference Childs2002; Vickers and Hogg Reference Vickers and Hogg2006). This has led to much debate about the place of art in sonification. It has been argued that overly artistic sonifications might miss the point of by opting for expression over the revelation of data features. A debate between utilitarians and aestheticians has run since the field's inception, characterised by a misreading of aesthetics as an exclusively artistic pursuit where in reality art and aesthetics are not synonymous (Barrass and Vickers Reference Barrass and Vickers2011). Aesthetics need not take away from the faithful communication of the data. Rather, they can provide new channels for sonification that fit our perceptual and cognitive faculties, better than the old model of linear, independent auditory dimensions.

In Reference Dewey1934 John Dewey demonstrated how meaning unfolds within the aesthetic dimension of human experience. To convey meaning is to shape this aesthetic domain to semantic effect. Mark Johnson (Reference Johnson1987, Reference Johnson2007) took the idea of aesthetics as the substrate of meaning and mapped the syntax (termed embodied schemata) by which that meaning is expressed. Lawrence Zbikowski (Reference Zbikowski2005) demonstrated how embodied schemata define our auditory experiences. As designers and creators of auditory displays, systems with the sole intention of meaningful communication, we cannot afford to overlook or relegate the very medium in which meaning unfolds: aesthetics. A semantically rich aesthetic framework could serve the sonic expression of data by offering more meaningful aesthetic channels along which to sonify data.

5. The Aesthetics of Embodied Music Cognition

For an auditory display to be broadly accessible it must appeal to some commonly shared aesthetic framework. However, aesthetic values tend to differ from person to person. For example, musicians routinely outperform non-musicians in interpreting auditory displays (Walker and Nees Reference Walker and Nees2011). In 1781, Kant proposed that the universe provided its own aesthetics and logic of which the embodied human had but a limited access (Kant Reference Kant1929). Today, in stark contrast, embodied cognition researchers argue that the human body provides its own aesthetics and logic. These are common to all similarly embodied organisms although not universal in the Kantian sense (Johnson Reference Johnson2007). It is to such a broadly inclusive embodied aesthetics that auditory display must look.

In Conceptualizing Music: Cognitive Structure, Theory, and Analysis Lawrence Zbikowski gathers together concepts and theories from across the embodied cognition corpus which ‘are visible at every turn in our encounters with music’ (Zbikowski Reference Zbikowski2005: 333) and applies them in musical analysis. His framework provides an alternative interpretation of sound, music and meaning. These cognitive processes imbue our auditory experiences with meaning (Zbikowski Reference Zbikowski2005: 328), shaping our understanding of rhythm, pitch, tempo and timbre. This meaning-making unfolds itself on the aesthetic level, and aesthetics are critical to conceptual meaning and reason. Experientialists argue that aesthetic experience is embodied meaning-making at its most potent. Johnson (Reference Johnson2007: 261) describes aesthetic effect as the heightened stimulation of our embodied meaning-making capacities.

Embodied schemata (Johnson Reference Johnson1987) are the basic units of cognition. These dynamic, gestalt-like frameworks are derived from the recurrent perceptual patterns encountered in daily life. By virtue of having similar physical bodies, entire populations use similar embodied schemata in their meaning-making. They provide the logical and aesthetic syntax by which auditory experiences are interpreted (Zbikowski Reference Zbikowski2005; Johnson Reference Johnson2007).

In ‘conceptual metaphor theory’ (Lakoff and Johnson Reference Lakoff and Johnson1980) embodied schemata are mapped to lend familiar structure to new perceptual and cognitive domains. This cross-domain mapping is an essential mechanism of cognition and provides a basis for building conceptual models and networks. Metaphorical mapping allows for the perception of, for example, movement in music by mapping schemata from bodily experiences to the musical domain (Johnson Reference Johnson2007; Zbikowski Reference Zbikowski2012).

Conceptual blending describes how meaning emerges from the combination of multiple spaces (perceptual and cognitive) so that elements of the input spaces are integrated and elaborated (Fauconnier and Turner Reference Fauconnier and Turner2002). It has been used to account for musial features like ‘text-painting’ (Zbikowski Reference Zbikowski2005). Blends draw from conceptual domains that are rooted in embodied schemata. Outside of the work of Zbikowski (Reference Zbikowski2005), and Fauconnier and Turner (Reference Fauconnier and Turner2002), the process has also been studied in the context of human–computer interaction (Imaz and Benyon Reference Imaz and Benyon2007) and education (Tolentino, Birchfield, Megowan-Romanowicz, Johnson-Glenberg, Kelliher and Martinez Reference Tolentino, Birchfield, Megowan-Romanowicz, Johnson-Glenberg, Kelliher and Martinez2009).

Prototype theory (Rosch Reference Rosch1999) states that members of mental categories are graded on a scale of prototypicality where the most prototypical member acts as a representation of the entire category. We then reason about categories in terms of their prototypical members. Music is understood at a basic level in terms of graded categories of musical events. Motive-based, rhythmic and atonal structures exhibit categorical structuring where prototypical members and their graded counterparts give rise to a unique syntax within a piece of music (Zbikowski Reference Zbikowski2005). These prototypes tend to be gleaned from the first-hand embodied experiences of day-to-day life. All category members, prototypical or otherwise, derive from embodied experience and are rooted in embodied schemata.

These competencies come together in conceptual models, which are aesthetic, knowledge structures comprising multiple cross-mapped concepts around a common theme (Zbikowski Reference Zbikowski2005). They determine the meaning and aesthetic nature of what we hear. Aesthetic experiences, such as the quality of a certain timbre, perceived movement in a specific section or proximity of a perceived source, are all formed through the interaction of these cognitive competencies. Even one's perception of pitch relies on a conceptual model that maps the schemata from bodily experiences of ‘up’ and ‘down’ onto pitch scales thus rendering it meaningful (Zbikowski Reference Zbikowski2005).

The crucial mechanism by which sound patterns become aesthetic experiences is the cross-domain mapping of embodied schemata. Embodied schemata provide the perceptual basis by which metaphors, blends and category members are built up into the meaningful conceptual models that define aesthetic experience and meaning-making (Zbikowski Reference Zbikowski2005). They provide the aesthetic and logical content on which there rest of cognition rests.

6. Towards an Embodied Cognitive Aesthetics for Auditory Display

Sonification has borrowed many concepts and insights from its sister field, visualisation. In visualisation designers do not map data to random visual dimensions, such as line height or thickness, as sometimes seems to be the case in sonification. Data are represented through the organisation of the features of a visual symbol in keeping with an overarching syntax. In a pie chart, for example, a sectored circle is used as a visual symbol. Features of the circle (the sectors) are organised to represent features in the data. It has its own syntax where the percentage area of each sector corresponds to a point in the data set. The syntax relates the data to the symbol. Without this overarching syntax any relationship between the data and the symbol would be arbitrary. In auditory display, the syntax that relates data to audio is referred to as the mapping strategy.

The human body provides its own logic and aesthetics in the form of embodied schemata. These can be called on to inform mapping strategies for sonification. Data need not be directly mapped to entangled auditory dimensions. Instead they can be mapped at the level of embodied schemata. Dimensions within an embodied schema can be represented in sound, allowing for the expression of data as sonic changes along these dimensions. This approach has merit beyond offering possible solutions for the mapping problem. Embodied schemata are acquired in early childhood and do not require learning. They are subconsciously recognised when encountered in perception and their meanings are intuitively felt (Johnson Reference Johnson1987; Hampe and Grady Reference Hampe and Grady2005). Mapping strategies based on embodied schemata can be understood without the need for much learning. This makes them critical to the design of intuitively familiar auditory displays. A cohesive account of embodied schematic mappings would be of benefit to the further development of intuitive auditory displays and to solving the mapping problem. Preliminary research in this area has been promising (Roddy and Furlong Reference Roddy and Furlong2013a).

In parameter mapping sonification (PMSon) the relationship between sound and data is often arbitrary. The task of relating data to sound can fall solely to the mapping function. This is not the case with auditory icons, which mimic real-world sounds to represent data. Auditory icons are chiefly used in human–computer interaction to add sonic feedback to a system. They are not typically used to sonify multi-dimensional data, as they do not provide individual auditory dimensions to which data can be mapped. They do provide higher aesthetic and cognitive dimensions which data can be mapped to. In recent years, a theory of ‘auditory affordances’ (drawing from ecological psychoacoustics) has been recommended as a design framework for such mappings (Brazil and Fernström Reference Brazil and Fernström2011). Preliminary research into the links between conceptual metaphor and auditory icons has shown positive results (Brazil and Fernström Reference Brazil and Fernström2006). Parameterised auditory icons (icons that express data through sonic changes) with embodied schematic mapping strategies have been used to sonify rainfall data (Roddy and Furlong Reference Roddy and Furlong2013b), as more complex everyday sounds lend themselves well to embodied schematic mappings. Further research in the use of embodied auditory icons for multivariate data will help to develop a better comprehension of the role that complex sounds can play in an embodied cognitive aesthetics for auditory display.

An understanding of how best to map specific embodied schemata for specific sonification tasks would be of great value. This knowledge would be invaluable in the creation of a repository of task-oriented, reusable mapping strategies in auditory display, such as that proposed by Degara, Nagel and Hermann (Reference Degara, Nagel and Hermann2013). A deeper understanding of the workings of a users perception of auditory symbols in terms of prototype theory would also be of great use to the design of meaningful sounds. The first step along this road is the recognition of aesthetics as critical to the process of sonification itself beyond merely cosmetic concerns. The second will be the recognition of embodied meaning making as critical to auditory display aesthetics.

The point of an embodied aesthetic framework is to enhance the current design methodologies in auditory display. It should always be applied alongside a chosen sonification methodology. It is not possible, or even preferable, for an aesthetic framework to replace these methodologies outright. Aesthetic concerns must serve the overall function of the auditory display. In the same way designers look to auditory scene analysis (Bregman Reference Bregman1994) or ecological psychoacoustics (Neuhoff Reference Neuhoff2004) to inform their work, it is suggested here that they also consider what embodied cognition has to offer. It is best suited to creating semantically rich auditory displays. An awareness of embodied meaning-making in design facilitates the creation of artefacts that fit our meaning-making capacities. We envision two scenarios for its application. Firstly, frameworks of this type should be considered when a designer wishes to convey a rich sense of meaning. Secondly, an embodied approach should be considered when the designer wishes to circumvent the mapping problem.

7. Example

The mapping of data along the embodied schematic auditory channels has been referred to repeatedly in the current article. What would such a process actually look like and how can it be achieved? Firstly, a data-relevant embodied schema must be chosen to provide the sonic framework for the sonification. One example uses the ‘centre–periphery’ schema to structure a sonification of rainfall data (Roddy and Furlong Reference Roddy and Furlong2013b). This schema, and the ‘near–far’ sub-schema, describes the conceptual structure common to bodily experiences of centrality and periphery. The schemata used are said to be implicit in the comparison and evaluation of states in a dynamic situation (near–far) and the treatment of important information as being centrally located in space (centre–periphery) (Lakoff, Espenson, Goldberg and Schwartz Reference Lakoff, Espenson, Goldberg and Schwartz1991). A short parameterised auditory icon of rainfall is then be modulated in keeping with the logic of this schema. There are two data points to be communicated: days of the week, and probability of rain. The seven days of the week are represented using amplitude envelopes structured in terms of the source–path–goal schema. This schema has been implicated in our understanding of time (Lakoff and Johnson Reference Lakoff and Johnson1999). Audio fades in to represent the beginning of a day, it continues for an explicit time period, and then fades out. This pattern is repeated seven times, to represent each day. Modulating the distal cues of the rainfall icon in the auditory stage, in accordance with the ‘near–far’ and ‘centre–periphery’ schemata, represents the probability data. The greater the chance of rain the closer the rain sounds to the listener.

The approach articulates the latent relationships that exist between the data points within a data set. It is not intended to convey discrete data points, which tend to require some form of symbolic mediation. This angle is pursued in the interests of answering the call for more ‘meaningful’ data to sound mappings (Neuhoff and Heller Reference Neuhoff and Heller2005; Walker and Nees Reference Walker and Nees2011). This approach places minimal learning requirements on the user by employing an already familiar syntax and symbol.

Design approaches of this type call for the use of phenomenological research methodologies for analysis and evaluation. Barrass and Vickers (Reference Barrass and Vickers2011) discuss in greater detail the need for phenomenological methodologies in evaluating auditory display aesthetics before suggesting IPA (interpretative phenomenological analysis) as one such tool. The example presented here used a co-operative evaluation methodology borrowed from human computer interaction. Future explorations of the aesthetic dimension in auditory display would benefit greatly from a standardised phenomenological evaluation methodology.