2. Scientific Background

The manner in which rocks break may influence the earthquake potential of seismic faults as well as control the dynamics of rockslides and glaciers. Numerical simulations of faulting provide a powerful way to visualise what may be happening inside a fault as it moves, whereas in real faults this information is hidden deep in the Earth. Mair's fault fragmentation models are 3D discrete-element based (DEM) models consisting of particles that push on their neighbours, implementing a realistic fracturing of granular debris during fault slip. The models involve aggregate ‘grains’, made up of 10,000 particles stuck together into sub-grain clusters with breakable bonds. Figure 1 visualises this process. As the walls of the models are driven at a given stress and the simulated faults slip, these aggregate grains break and the granular debris evolves in shape and size in a somewhat natural way (Abe and Mair Reference Abe and Mair2005; Mair and Abe Reference Mair and Abe2011). These simulations probe the sliding of a mature fault and reveal how granular fault debris evolves and also affects future sliding (Mair and Abe Reference Mair and Abe2008; Abe and Mair Reference Abe and Mair2009).

Figure 1 Numerical simulations: the granular debris model is shown for four increments of increasing deformation at timesteps (a) 2, (b) 20, (c) 40 and (d) 50. The granular fault material is gradually broken up as the plates (top and bottom) are compressed together, then sheared as indicated by the arrows.

To simulate the conditions a real fault rock would ‘feel’ at depth in the Earth, laboratory experiments are conducted where rock samples are literally squashed under carefully controlled loading conditions in a high-pressure rock deformation apparatus (Figure 2; Schubnel, Thompson, Fortin, Gueguen and Young Reference Schubnel, Thompson, Fortin, Gueguen and Young2007). Mechanical information, such as the (imposed) stress and resulting strains (changes in shape or volume), are continuously recorded during the experiments, and post mortem observations of the final broken samples (using CT scanning or microscopy) are carried out. This approach helps to build a picture of damage evolution and final 3D fracture systems created, respectively. Although it is harder to ‘look inside’ such an experiment than a model, it is possible to ‘listen’ to high-frequency ‘acoustic emissions’ – burst type acoustic signals (in the KHz to MHz range) produced by microscopic fracturing and sliding events deep inside the rock. These signals are recorded using up to 24 ultrasonic transducers distributed on the 3D surface of rock sample inside the pressure apparatus (Figure 2). Recording the waveforms associated with these tiny acoustic emission events at many sensors, and comparing relative amplitudes and times of arrival, allows the researchers to determine the magnitude and spatial location of each individual micro-fracture event. Through time, this data reveals physical microscopic changes (fracturing and sliding) within the rock sample due to the imposed stresses, and yields valuable insights into rupture propagation. The intense acoustic activity in an experiment (when the entire sample breaks) lasts only 0.15 seconds, but the extremely high sampling frequency (4 MHz) captures data points every 0.25 microseconds.

Figure 2 Laboratory rock deformation experiments: (a) cylindrical rock sample is sleeved in a protective rubber jacket, with a set of 16 piezoelectric transducers to measure acoustic emissions (arrows) attached to its sides then inserted in rock deformation apparatus (b) that applies fluid pressure to the sides of the sample (to simulate depth in the Earth) and additional load along the axis of the sample (arrow) by servo-control loading platens. Experiments were conducted in the laboratory of Alexandre Schubnel at École Normale Supérieure, Paris. (Photographs K. Mair.)

3. The Crush Audio system

So that listeners can experience the intricacies of the inherently 3D process of rock deformation from the inside, it was most interesting to work in 3D sound using an appropriate sound-field recreation technology. To allow a single primary listener to interactively explore the sound space and additional listeners to experience this exploration, sound needed to be heard over both headphones and loudspeakers. At the time we developed our collaboration it was rare to find examples where spatial audio was accurately addressed in sonification. Recent work from the Seismicsoundlab (Holtzman, Candler, Turk and Peter Reference Holtzman, Candler, Turk and Peter2013) has explored the audification of global seismic waves through panning sounds over 16 speakers, where the sound location corresponds to the relative positions of the seismometers. For our work in 3D, ambisonics was chosen since it allows us to render for loudspeakers without pre-specifying a specific number or arrangement, and for headphones using binaural decoding. We chose to encode information in higher-order ambisonics (HOA), allowing us the options of HOA and first-order loudspeaker decoding as well as binaural rendering using the KEMAR HRTF set (Gardner and Martin Reference Gardner and Martin1994) implemented in Harpex (Berge and Barrett Reference Berge and Barrett2010). (For a basic introduction to first-order ambisonics and HOA, Hollerweger Reference Hollerweger2008 is recommended reading).

Furthermore, ambisonics permits either pre-encoding the spatial sound scene and decoding it later in listening, or encoding and decoding in real-time. Both methods were used, depending on the data. For smaller amounts of information, such as a reduced set of mono points, it is possible to encode and decode in real-time. This is useful in interactive contexts as the listening point can be constantly updated with respect to the location of the sound point and an accurate interactive spatial relationship maintained. However, for large amounts of information (hundreds of points in short time frames or clouds of sounds) it is more practical to pre-encode the spatial information. The disadvantage of pre-encoding is the limit on interactive accuracy: once encoded, the relationship between sound point and listening point cannot change as in reality. In this work, a compromise was made where scenes heard from five possible listening positions were encoded – one in the centre and one on the edge of the ambisonics space at 90 degrees separation. These scenes were cross-faded respective to the listening point. Prior to developing the interactive interface for Crush (see section 7.1) a simple mouse interface was used to navigate through or transform the ambisonics scene.

4. General mapping and scaling needs

Capturing the patterns and processes already embodied in Mair's work was the starting point for Barrett's composition. The laboratory experiments and numerical simulations presented different types of data requiring different approaches to sound. If we consider sonification by audification (Dombois and Eckel Reference Dombois and Eckel2011), a time-ordered sequential data stream of thousands of points was directly converted to sound. This method was used to sonify Curiosity's 2011 landing on Mars, compressing 20 minutes of signals into a 19-second clip. We used a similar approach to sonify Schubnel's acoustic emissions. In contrast, the numerical simulations were approached with mapping-based sonification, where more choices are needed in addressing thousands of data points.

The three scientists Mair, Abe and Schubnel explore fracture processes operating at the micro-scale – that is, microns to millimetres that sum up through time to affect system behaviour at a much larger scale. The scientific source data consists of:

• • thousands of fracture events occurring at a given time, with a given magnitude and spatial location;

• • the spatial motion of granular debris; and

• • ultrasonic sound signals associated with individual fractures.

The initial challenge was capturing the complexity of the deformation processes rather than balancing art with science. Scaling the relationship between scientific data ranges and audio parameters was a critical stage in both music and sonification. The author's own programmes (unpublished), Vspace (Furse Reference Furse2000) and MaxMSP (Cycling ’74 2011) were used. All sound parameters were scalable with the aim of enhancing variation and pattern on a perceptual level, where even the smallest variations could be made audible.

5. Sonification of numerical simulations

The numerical simulations comprised 500,000 particles (c. 800,000 breakable bonds) producing a vast dataset of fracturing events during 2 million time periods. It was essential to reduce this dataset before considering how to approach the sonification. The researchers also needed ways to visualise their results and had hence extracted ‘snapshot’ data from the model in 200 evenly spaced time-intervals that span the simulation. This new dataset became the primary source for sonification. There were nevertheless still over 200,000 multi-parametric events, and the scientists advised on the key parameters to consider. Figure 3 shows an example of fracture events occurring for a single time interval. An example extract of the data is shown in Table 1, where x, y, z show the position of the broken bond – in other words, the fracture.

Figure 3 A snapshot of fracture locations extracted from a single time interval in the numerical simulation (shown in Figure 1). Individual fractures are shown as small black discs. The remnant debris of two broken aggregate grains are shown in semi-transparent grey.

Table 1 Fracture position and magnitude extracted from time-step 100

To gain a basic understanding, the whole data set was sonified in a simplified fashion using the following scalable parameters:

• • Individual sounds were mapped to individual data points. Two input sound types were tested: sine tones of a specified duration and short acoustic sounds.

• • Spatial information (x, y, z) in the data was used as point locations in the ambisonics synthesis. For headphones the full 3D model was used. For loudspeakers, vertical energy was projected into the horizontal plane.

• • Z also determined the frequency of the sine-tone or the pitch shift of the input sound file.

• • Fracture magnitude was mapped to volume.

• • Time was mapped to time.

It was immediately clear that data chopped up into 200 time-steps would be less than ideal for the sonification because artificial ‘time chunking’, in which all events within one time-step occur in unison, aurally misrepresented the true behaviour in the system in which original events were distributed in time. As a solution, the events were randomised within the duration of each time-step. This added uniform noise that disguised the artificial time-chunks. Outside the duration of the time-step, interesting patterns began to emerge.

After a few tests runs, the following scaling best illustrated the data in sound:

• • time scaled to a total duration of 10 minutes (3 seconds per time-step);

• • sine tone input sources of 3-second duration (breaking bonds themselves have no timeframe in this context) and acoustic sounds with a clear attack;

• • spatial scale of 20 × 20 × 20 metres (with volume and filter room model);

• • pitch shift range of two or four octaves for acoustic sound input or 100–10,000 Hz for sine-tone input;

• • volume scale from 0 to 1 (where 0 is silent and 1 is maximum volume before clipping) for each event based on magnitude and then normalisation before the complete sonification was written as a sound file.

The results showed a chaotically complex initial phase followed by exponentially decreasing fracture activity distributed over the 3D space. Sound example 6 plays the first 30 seconds of a 10-minute sonification using input from Sound example 7.

5.1. Numerical simulations and musical development

After the strict scientific approach, musical characteristics were aurally discovered as a secondary effect emerging from the system. Ballora (Reference Ballora2011) and Ben-Tal and Berger (Reference Ben-Tal and Berger2004) also explore musical issues, blurring whether choices are based on scientific principles or on aesthetics. In Aftershock, musical choices rather aimed to maintain a strong scientific connection to the data. Four areas were targeted for further work:

• • Highlight interesting emerging features with data reduction and selective data processing.

• • Highlight emerging musical ideas by any means.

• • Refine scaling ranges.

• • Explore new mapping rules.

5.1.1. Data reduction focusing on fracture magnitude bands

The first time-step (where a substantial amount of fracturing took place) was removed and treated separately. Data in the other 199 time-steps was filtered into magnitude bands. Data in the lowest band was removed. Events in the other bands exhibited clearer trends over the long term (outside the 3-second duration of the time-step). Each band was then mapped to a unique sound source consisting of a short attack made from Barrett's own ad hoc rock deformation recordings. Below are some sound examples for the first 10 seconds/5 time-steps:

−<0.7 (data removed)

0.7 −<0.75 (Sound example 8)

0.75 −<0.8 (Sound example 9)

0.8 −<0.85 (Sound example 10)

0.85 −<0.9 (Sound example 11)

0.9 −<0.1 (Sound example 12).

Sound example 13 is the opening 90 seconds of a 10-minute sonification selecting sound (from various magnitude bands) for its musical value and combining sine-tones with acoustic sources as inputs. This was the default opening to Crush (section 7.1).

5.1.2. Emerging ideas: particle paths

Single particles (Figure 4) were extracted from an earlier numerical simulation (where smaller forces and hence less fracturing occurred) and their activity traced in sound and space. The particle path became a musical focus. For each path, one mono input sound file was generated using a granular synthesis approach in which grain size and density were controlled by the particles’ proximity to the centre of the spatial data set. This central point also served as the location of the virtual microphone in ambisonic rendering. Close proximity to this centre resulted in a ‘gritty’ texture, while further away gave a smoother texture. The result was a spectral-temporal quality appropriate for audibly identify a change in space, as well as enhancing distance perceptual cues. The trajectory was spatialised in an ambisonics room model 50 metres in size. As before, the z co-ordinate (height) was also mapped to pitch transposition (Sound examples 14–20 described below use this method).

Figure 4 Numerical simulations: a series of single particles are selected and their paths traced during a simulation.

5.1.3. Refine scaling ranges

In space, one particle may traverse only a small absolute area. However, relative movement is a key player that is useful to enhance in two ways: scaling the space to that of the particle's movement, and centring the virtual microphone on the particle motion rather than on the centre of the complete dataset. Both techniques highlight relative motion (i.e. changes) rather than absolute distance. Furthermore, in the temporal domain, if sonification of the complete particle track was scaled to 10 minutes, as was the case for the complete data set, the change in spatial position through time would be too small for us to hear. Scaling the complete track between 20 and 100 seconds highlighted temporal-spatial dynamics in sound. Movie example 2 visualises the motion of six particles over 20 seconds.

Sound example 14 sonifies particle 10631066 over 20 seconds using a virtual microphone position in the centre of the complete dataset space. Sound example 15 scales the space to that of the particle's spatial motion and places the virtual microphone position in the centre of this motion. In Sound example 16 all six particles are mixed together. The spatial dimension and the x-y microphone location are based on the average of the six particles, while the z microphone location is placed centrally for individual particles. In this way the vertical distance away from the microphone is reduced, enhancing the resulting angular motion in the x–y plane (and the z axis is already represented by pitch shift). Sound example 17 stretches time to 100 seconds. These time developments are useful for sound-art, allowing textures and timbres to unfold at a slower rate.

5.1.4. Explore new mapping rules

The particle is the smallest unit in the model and cannot ‘break’ to create events onto which sound can be mapped, but their path is strongly controlled by the breakup of their parent grains and interaction with neighbouring fragments and particles. The relation between particles and clusters (sub-grain-sized fragments) is interesting, and cluster changes were treated in a similar way to the fractures in the complete data set. In the selected numerical simulation, the cluster to which a particle belonged changed mass only a few times and so data reduction was unnecessary. Sound example 18 mixes the fracturing of 30 particles’ clusters at a 1-minute time scale using input from Sound example 19. Interesting sound developments were explored by simply changing the input sound type and pitch scale.

When the cluster changes are played in parallel to the particle trajectory we hear which parts of the particle motion are influenced by cluster fracture. This approach adds interesting musical details. Sound example 20 plays the trajectory of particle 4531543 at 20 seconds time scaling with cluster breakages mixed in.

6. Sonification of acoustic emissions data

From the laboratory rock deformation experiments, Schubnel provided acoustic emission datasets containing the spatial positions (x, y, z) and magnitudes for 540 fractures as they occurred in real-time, along with the raw whole waveform ultrasonic signals, for three different rock samples (the samples were initially intact and gradually broke as deformation proceeded):

• • sandstone (from Fontainebleau, close to Paris, France);

• • granite (from La Peyratte, close to Poitiers, France); and

• • basalt (from San Miguel Island, Azores, Portugal).

Figure 5 (and Movie example 3) shows behaviour approaching rupture and eventual sample failure in a Fountainbleau sandstone.

Figure 5 Laboratory induced acoustic emission: (a) sketch (side view of rock sample) illustrates how the ultrasonic sound from a single microfracture event travels to and is recorded at multiple transducers; (b–d) locations of acoustic emission events plotted in 3D space as a function of time (i.e. accumulated loading) for a sample of Fountainbleu sandstone. The outline indicates the edges of the rock sample. Snapshots are shown. (e) The final image shows all the accumulated fracturing events during the entire experiment. (Images courtesy of A. Schubnel.)

Laboratory-induced acoustic emissions and numerical simulations explore two different aspects of fault zone processes respectively: (a) the birth of a fault (dominated by compressional and tensile forces), and (b) the sliding of a mature fault containing fault debris (where, macroscopically, shearing forces dominate, but tensile forces also act). However, there is another fundamental difference: a laboratory experiment captures real fracturing whereas a numerical simulation models fracturing. This difference influenced Barrett's method of working with the laboratory-induced acoustic emissions: the investigation began with real sound. The whole waveform ultrasonic recordings (made during the experiments) were transposed into the audible range using a standard down sampling (tape transposition). From the audible range transpositions differences between the fundamental behaviour, rather than the audio spectrum, of the different rock samples could be clearly heard. All three samples showed increasing acoustic activity with time; however, each rock displayed quite distinct acoustic behaviour leading up to rupture and sample failure. This is a direct reflection of the distinct mineral grains, their arrangement and bonding in each rock. Sound examples 21, 22 and 23 play sandstone, basalt and granite waveforms respectively, transposed down 10 octaves. These sources were a guide for sonifying the laboratory data, which was approached similarly to the numerical simulations, but where two new classes of input sounds resulted in significant differences in the music:

• • Ultrasonic ‘art’ source recordings. As sound input, inspired by the lab recordings, Barrett made her own ad hoc ultrasonic recordings of rocks being crushed in a manually operated vice system. These recordings (akin to a rock-fall occurring at the surface of the Earth) allowed control over sound quality and spectrum. Ultrasonic recordings were made with an Avisoft-Bioacoustics CM16/CMPA-P48 transducer and a Fostex hard-disc recorder at a sampling frequency 196 KHz. Samples of slate and earthenware composite were chosen as they produced interesting sound at lower stresses than in experiments and at frequencies below 90 KHz (most probably due to differences in the microstructures of the materials). Normal condenser microphones were also used.

• • Transposed scientific ultrasonic recordings. The transposed scientific ultrasonic sound recordings (mono, continuous) were spatially mapped to each rock sample dataset consisting of discrete points, in a continuous ‘dot to dot’ fashion. Sound file duration and the data timeline were scaled to synchronise. The scene was encoded with the virtual listening position at the centre of the data space. In Sound example 24 we hear that the result is a spatialised sound jumping to spatial-magnitude points coinciding with the dynamic acoustic changes of the original sound file. In this example, fracture events are also added as sound.

6.1. Acoustic emission data and musical development: a process of abstraction

The focus of the numerical simulations on the evolution of a mature fault gouge results in initial intense fracturing, followed by decay and enhanced relative motion of the particles. This leads to rich patterns where the resulting sonification may appear more abstract than the simple intuitive idea of a rock breaking or of brittle fracturing. In contrast, the laboratory acoustic emissions build up gradually then accelerate into a dynamic rupture followed by a rapid decay, where both original sound and musical results of the data sonification are strongly akin to an intuitive idea of a rock breaking. It therefore felt appropriate to explore musical abstractions of the laboratory acoustic emissions sonifications to balance the more abstract results of the numerical simulations sonifications.

6.1.1. Abstraction through sound transformation, mixing and juxtaposition

So far, the input sounds to the sonification have consisted of short attacks mapped to fracture events. If, instead, the input sounds are lengthened to reveal their own internal detail, a more complex pattern will result. Further, if the input sound contains a noticeable pitched centre or fundamental frequency, the transpositions of this sound (where pitch is mapped to vertical displacement), will lead our listening to detect pitch structures and discover musical phrases as more evident than noise collages. Sound example 25 is the granite acoustic emissions data sonified at a 33-second duration, with one such input sound.

Previous audio examples have illustrated the different acoustic properties of three rock samples and how fracturing occurs through time in both scientific ultrasonic recordings and in the sonification of the acoustic emissions data. By mixing the results from different rock samples sounding over different temporal durations, a compositional approach emerged: setting one material in counterpoint to another produced a heightened musical climax and a new musical entity out of originally separate materials. This approach is illustrated in Sound example 26, which is a 3′40′′ mix of sonification and scientific ultrasonic recordings.

6.1.2. Abstraction through the active listener

When all spatial fracture data from the acoustic emissions is plotted in space it creates a trace where the timeline of the fracture process is removed. From this trace, 40 of the highest magnitude fractures were selected, projected onto a 2D horizontal plane rotated 90 degrees (so that the vertical view in Figure 5 becomes horizontal) and scaled to a dimension of 20 × 25 metres (Figure 6). Each point in Figure 6 is allocated a unique sound that corresponds to its magnitude, where louder, longer or richer sound spectrums are mapped to higher magnitude fractures. These sounds loop continuously, their locations encoded and decoded in real-time ambisonics. The result produces a cloud of fracture points that constantly ‘tick over’. The composition only exists when the listener interacts spatially: if standing in one location, for example the centre of location A, a listener will be almost on top of a fracture point and hear a loud and bright repeating sound with other sounds within ‘earshot’ appearing further away. By exploring the space, the listener then experiences the relation between fracture points where spatial information is constantly updated with respect to the listening location: for example, if they move to location B. Sound example 5 interactively explores the granite projection in different directions (using the mouse controller).

Figure 6 Sound points in an ambisonics virtual space seen from above.

This new way of expressing the acoustic emissions rupture was developed further in Fractures Frozen in Time (Frieze 1–3). In this work, an interactive ‘walk through’ for each of the three rock samples was fixed to a recording. The recording was then decoded to five channels and distributed over 50 miniature loudspeaker elements fixed in a random and even distribution along a 10-metre wall (Figure 11). The sounding result is a development of that heard in Sound example 5.

7. Combining two scientific experiments into one artwork and Crush

The discussion so far has described scientific sonifications followed by musical developments. In reality, many results from the initial sonification work were already coloured by artistic choices to explore musical expressions of process, form and internal structure. In Crush, although scientifically incorrect to do so, elements derived from both scientific experiments were juxtaposed. We hear the different processes set against each other, yet a sense of convergence was sought in practical approaches that addressed numerical data and a shared pool of source sounds. Departure from scientific accuracy had however already begun: musical abstractions were achieved by specific input sound selection, sound transformation and by scaling parameter ranges for musical rather than scientific goals.

Unlike sonifications played in isolation, Crush does not exist by ‘pressing play’. It is an interactive spatial audio installation that only exists when there is an exploring listener, at which point being a spectator or passive listener also makes sense.

7.1. Crush as sonification and sound-art revealed through interaction

As an interactive 3D sound space, Crush-3 requires two interfaces: one for when physically stationary and exploring ‘virtually’ and one for when physically moving in the installation space. For both interactive methods the ambisonics sound-field is translated and rotated in real-time with respect to the new listening point. For physical interaction, we use a low-cost and low-tech motion tracking system, designed by Berge and Barrett (unpublished 2009) that allows users to physically navigate through the 3D sound composition. A head-mounted unit, worn by the ‘explorer listener’, contains 3D accelerometers, gyroscopes and an infrared camera. Seven infrared light constellations, each of a unique 3D geometry, surround the space. As the explorer moves, the lights come into view of the camera (Figure 7) and actively recalibrate drift from the accelerometers and gyroscopes. For further accuracy, triangulating the 3D geometry of the light patterns provides the computer with a second set of positional information. Motion data are sent to a computer over Bluetooth and processed to render the explorer's position and direction of view. This information modifies the spatial sound image. For the explorer listener wearing the headset, sound is rendered using HRTFs over wireless headphones. Image rotation on the headphones is matched with the rotation of the sound on the loudspeakers. Using headphones allows the explorer to hear a more accurate sense of direction making their navigation intuitive, as well as solving spatial problems when moving outside the ambisonics loudspeaker sweet spot. At all times the explorer is free to move within the space defined by the LEDs, their listening location updating the audio spatial perspective. In the interplay of listener, space and sound, the work reveals itself through motion. For other listeners, sound is decoded over the loudspeaker array.

Figure 7 Crush motion sensor helmet and wireless headphones.

To explain how the audio/musical elements of Crush tie together we can consider each stage of the interaction:

• • Stage 1: With no active interaction, the full data set of the numerical simulations plays. This sound-file is the 10-minute time-scale duration, five-magnitude band sonification explained in section 5.1, mixed with the earliest sine-tone experiments explained in section 4.2. The sound was pre-encoded in ambisonics with a central virtual listening position (Example 7).

• • Stage 2: Immediately the sensor helmet or mouse is moved, seven short mono sounds loop continuously and stage 1 now becomes the background. Each sound is given a unique spatial location computed in real-time so it remains stable when the explorer listener moves. These sounds were created from the scientific ultrasonic recordings, transformed into seven variations (processed versions of Sound examples 21–24).

• • Stage 3: The exploring listener is asked to aurally navigate and physically move to one of the seven looped sounds. With close proximity, a more dramatic change is triggered: the numerical simulations’ sonification and the seven looped sounds fade out; the acoustic emissions data or the numerical simulations particle trajectories begin (examples from section 6.1and 5.1.1). Some of these new materials have been ambisonically pre-encoded from five listening positions, and by moving (see section 3) the listener ‘cross fades’ into a new spatial perspective. Other materials may consist of real-time ambisonic encoding, achieving more accurate spatial relationships, such as the approach explained in section 6.1.1. Whichever type of material is triggered, stage three marks a greater musical exploration and departure from more precise scientific representation.

• • Stage 2b: Stage 3 has a fixed duration derived from the material durations, after which stage two resumes with one exception: the sound that ‘released’ stage 3 is now silent, such that it cannot be triggered a second time until all of the other six points are explored.

An Internet link to an 18′30′′ real-time interaction performance of Crush rendered to stereo is available (Barrett Reference Barrett2012a).

7.2. Environmental constraints and exhibitions

The physical motion interface was developed for Crush-1 at the 29th Nordic Geological Winter Meeting January 2010 at the Radisson Blu Scandinavia Hotel in Oslo. During this short exhibition, it transpired that the system required supervision for optimal fitting and use.

Crush-2 was installed for three months, summer 2011, in the Norwegian Technical Museum in Oslo at the EU-funded COST IC0601 Action project Sonic Interaction Design (SID). This exhibition needed to withstand the unsupervised interaction of hundreds of school children. In this setting, a 3D mouse controller was deemed the best interaction method; it controlled motion, direction and rotation, and a computer display showed a simplified bird's-eye view of the interactive area and the track explored, located in the centre of the space. Sound was decoded over an octagon loudspeaker array. Although interacting listeners were located in the sweet spot they also wore headphones to aid concentration in an uncontrolled external audio environment (Figure 8).

Figure 8 Crush-2, Norwegian Technical Museum in Oslo, 2011.

Crush-3 was installed in Gallery ROM in November 2011 (Figure 9). ROM is a contemporary art and architecture gallery with staff present, making it possible to use the motion sensor system throughout the month of exhibition. The context also allowed performances, where the audience sat inside the loudspeaker array and Barrett explored the work using the mouse interface, making a real-time performance and turning Crush into a musical instrument.

Figure 9 Crush-3/Aftershock exhibition, Gallery ROM, Oslo, 2011.

8. The four elements of Aftershock: a second generation of juxtaposition

Aftershock combines Crush-3, Fractures Frozen in Time, Golondrina and Cleavage Plane in adjacent areas of one exhibition space where sound, but not physical areas, overlap. Aftershock can be set up in any similar space with only minor modifications in ambisonics decoding parameters.

Crush dominates the sound and ‘place’, while the other three works set at lower volume only play when triggered by a visitor. Crush-3 is therefore always sounding while the other works layer contrasting processes only periodically.

Cleavage Plane is the first work (Figure 10) when entering the exhibition. Its explicit process and real-time audio set a tangible framework for the invisible sound-worlds of the other works. Sharp and varied pings, cracks and plops play over a mono loudspeaker, creating a brittle and sparse foreground to the denser and more complex dynamics, temporal forms and spatial depths of Crush-3 occupying the main space. After the listener leaves the area, Cleavage Plane stops unless another visitor triggers a proximity sensor and maintains its activity.

Figure 10 Cleavage Plane/Aftershock exhibition, Gallery ROM, Oslo, 2011.

In Crush-3 we hear the complete juxtaposition unfold: processes spanning multiple time scales and spatial viewpoints, drawing on numerical simulations and laboratory experiments. If another visitor is already in proximity of Fractures Frozen in Time (Figure 11), this work will sound in the background, connecting conceptually and sonically to Crush-3 as explained in section 6.1.1. The work is a ‘slice’ or a ‘lower dimension’ in terms of space, sound, dynamics and process: the 3D of Crush becomes 2D in Fractures Frozen in Time, and full-range sound has been reduced to high-frequency rattles, tingles and soft, throbbing bass.

Figure 11 Fractures Frozen in Time/Aftershock exhibition, Gallery ROM, Oslo, 2011.

Between Crush-3 and Fractures Frozen in Time the visitor passes through Golondrina played over six loudspeakers hanging vertically in a stairway. Golondrina presents a new process and a new scale: blocks of stone falling from the roof and walls of a cave-collapse simulation with time scales of thousands of years condensed into 10 minutes. Each block triggers a sound. This simple sonification process creates less dense data and a linear accelerating dynamic, which in turn allows sound inputs of higher compositional detail. Golondrina's simple events and complex sound sources contrasts to the complex events and simple sound inputs of Crush.

As a whole, we hear a juxtaposition of temporal, spatial, dynamic and spectral domains, expressed through the juxtaposition of patterns and processes found in the scientific data. An Internet link to the video documentation of Aftershock can be found in the references (Barrett Reference Barrett2012b).

9. Discussion and Conclusions

Crush was created with two aims: for science (as public outreach) and for music as an exploration of new temporal structures, sound-worlds and as new material for composition. Collaboration with scientists produced a complete exhibition from one original concept – exploring 3D rock deformation from the inside – where the art and the science were sometimes in close proximity, at other times divergent, but maintaining a coherent picture.

In terms of public outreach, the three different versions of Crush were installed in three completely different contexts. For each version there were incremental improvements in sound, interaction design and real-time video elements (not discussed here). However, a significant difference was the audience. At the Nordic Geological Winter Meeting the audience was mainly geologists whose responses were strongly polarised: half enthused about the science–art connection, the others were unconvinced. At the Norwegian Technical Museum the context was an exhibition on sonification and interaction design. Children and adults knowing little or nothing about the science took time to explore the work despite competing distractions. In Aftershock at gallery ROM, visitors experienced physical interaction using the head-mounted tracking system. This controlled environment allowed them to explore the work in more detail.

In terms of compositional development, the sonification produced unique materials that could not have been created otherwise. The methods of mapping, scaling and data reduction allowed scientific techniques to inform the art through discovery, rather than any pre-stated artistic goals taking over the work. Intuitive compositional ideas then explored the musical connections between space, time, pitch and sound.