1. Beyond music?

A concierge, in the opening sequences of Jacques Tati’s Playtime (1967), who must announce Monsieur Hulot’s arrival to the bank operates a complicated electronic paging system installed on a wall. As he presses all the buttons on the board, weird, wonderful and humorous electronic sounds come to life. On one hand, the board itself has a dramatic persona now: it has a voice. On the other hand, the sound comes to us, almost accidentally, through a medium that has established an alliance with music since its inception: film. That sound can be organised and fixed on film is not a trivial fact. It has implications for both media, sound creativity and film. This article is an attempt to trace possible origins and perhaps even elements of an archeology of sound creativity on the fixed medium of film.

To reduce sound art to the creative practice we know as electronic or electroacoustic music is a mistake. The usage of sound for dramatic purposes, for representation and narrative goes beyond what we might conventionally term musical; in film-sound creativity, concerns with pitch, rhythm and timbre are secondary to meaning and reference. If we look at sound artist Walter Murch’s work for director George Lucas’ film THX1138, for instance, we find editing, placement and re-recordings of the actor’s voices where the main concern is to aid the telling of the story by fabricating cues that allow for a situated perception of sound. Among other techniques, transmitting voices via analogue radio broadcast and recording the result whilst fine tuning the reception frequency allowed both Murch and Lucas to augment the voices with synthesised harmonic content that made them more futuristically believable. The resulting sound helps render the textual plot in pure sonic terms. An example of this is the counterpoint of vocal transmissions during THX1138’s trial scene, where the use of layered voice-overs and radio tuning sounds effectively portray the helplessness of the character at the mercy of the legal machinery.

Can the term ‘musical’ describe sound constructions, such as these, properly? Arguably not. I would propose that we must think in terms of organised sound even beyond Varèse’s definition. We must think of the dramatic effect of sound through implied significations and not in terms of its isolated physical qualities (e.g. pitch and duration). Further, seeing as sound on film is there to coexist with the visuals, any understanding of its workings needs to be viewed in the context of intermedia; that is to say, coexisting media that conform, complement or contrast with one another (Cook Reference Cook2001). In opposition to the inner logic of pitch, rhythm and timbre found in musical discourse, sound on film cannot be evaluated without reference to the image.

The manipulation of meaning by sound on film is achieved through what Rick Altman calls ‘the story of the sound event’ (Altman Reference Altman1992). He discusses how a ‘spatial signature’ is carried by the sound that exhibits the trace of a particular hearing or listening situation. This is of course familiar to electroacoustic music theoreticians through the listening modes discussed by Schaeffer in Reference Schaeffer1966 in his Traité des objets musicaux. Yet Altman’s emphasis is precisely on the inference of referential audio cues from the sound object as opposed to the reduction of the sound event to its musical potential. Arguably, this difference is key in understanding how sound creativity on film generates different formal choices from sound creativity in electronic or electroacoustic music.

Altman’s views provide further understanding of the idea of source identification in the context of film, because he discusses the potential narrative content of a sound; how the sound conveys the context in which it was listened to and how this has representational value. For this reason, the choice of sound objects in film (even when they are not used as sound effects) is subject to their referentiality and not to their reduced characteristics in the Schaefferian sense.

2. A Plastic form

Film is possibly the first medium to efficiently accommodate sound. Since 1929, Russian drawn-sound composers Arseny Mikhaylovich Avraamov and Yevgeny Sholpo explored the plastic possibilities of sound on film (Davies Reference Davies2007). Around this time, in Germany, Fischinger and Moholy-Nagy, amongst others, also scratched and drew on celluloid as a synthesis method for sound and artistic expression. Not long after the first Russian experiments, in 1932, Fischinger wrote the following in an article entitled ‘Sounding Ornaments’:

In 1931 in the United States, and with characteristic Anglo-Saxon pragmatism, drawn-sound techniques were used by the film-sound crew working for Reuben Mammoulian in Jekyll and Hyde to help represent the first terrifying transformation of the mad scientist; drawn sound, noise and voice collage are all put to diegetic use with firm musical intuition. Instances such as these provide insights into the beginnings of sound creativity on film, and by charting them we can observe how early the foundations are laid for more abstract forms of composition with recorded and synthesised sounds. Further drawn-sound experiments would be carried out in the 1940s by Norman McLaren in Canada.

Earlier, in Germany, yet roughly parallel to the use of drawn-sound techniques, Walter Ruttmann presented his Weekend (1930), a de facto musique concrète work (also, perhaps, a very early instance of soundscape composition), roughly eighteen years before Schaeffer’s theorising led to similar (although much more abstract) sonic results. Weekend is a work of sound art on the film medium, and is normally ‘projected’ on a dark screen, which, thanks to its presentation, becomes a further intimation of acousmatic music. Whilst it cannot fairly be called a work of musique concrète, it should attract our attention in that it is the first piece to tell a story solely by editing sound without visuals, yet still being hosted on the film medium. Perhaps it could be more convincingly argued as a piece of soundscape composition as would be developed many years later by R. Murray Schafer (Schafer Reference Schafer1973), in any case it remains an interesting historical artefact that bears witness to the creative possibilities of the film medium.

Both of the approaches discussed above, drawn sound and soundscape composition, were, as far as we can tell, the precursors of fixed media sonic art or sound art. They are the first practical application of the film medium as a means for assembling and presenting sound art. Since there is no parallel development in electronic music independently of film, we could argue that the plastic manipulation of sound for artistic purposes actually begins on film and, further, that coexistence with images adds to sound composition a new set of creative concerns: synchrony, synchresis (Michel Chion’s term to define the relationship of unity surmised from seeing something in sync with a possibly unrelated sound), empathetic and anempathetic relationships as well as different narrative roles (Chion Reference Chion1994). These concerns also give rise to new treatments of musical form, as we have also mentioned earlier, where the referentiality of a sound justifies its placement within the compositional structure.

Photoelectric sensitive material, as in early sound film, and later 35 mm magnetic tape are media that allow for plasticity and thus naturally became a home for fixed sound art. This creative exploration of sound we refer to, independent of traditional musical instrumentality, results in what has been called, amongst several terms and to the inconvenience of many artists and composers, ‘sound art’. This practice of ordering recorded or synthesised sound events in a particular way that, mostly, may not follow ‘musical’ criteria, that prizes the referential content of sound over its acoustic characteristics, may have many names yet remains an identifiable trend; easier to recognise than to group under one umbrella term that will please all practitioners.

Two main approaches, then, that prefigure later practices in tape music composition seem to be present in sound creativity on film from the very start: the creation (synthesis) of new sounds, which results in ‘photoelectric’ music, and the use of recorded sound as a compositional tool. Both approaches predate musique concrète and elektronische Musik by several years.

3. Sound effects

Once the technology of sound reproduction on film was viably established, sound artists did not take long to emerge. Instances such as the Jekyll and Hyde (1931) transformation sequence of Dr Hyde into Mr Jekyll, the sound-designed growling of King Kong (1933) and the drawn-sound experiments of Russians and later Germans, showed how a whole gamut of possibilities were opened. From the more programmatic to the more abstract, different practices emerged showing an ancient dilemma of musical practice: internal versus external signification.

Sound crews in film, from the earliest use of recording and editing, have shown that their craft is greatly subjective. A sound ‘effect’ is not the actual sound of what we see on screen, so much as the kind of sound we expect to hear; the effect of the sound. Sound effects may also reflect cinematic listening perspective. They must follow the dramatic guidelines of film narrative to be useful, thus becoming involved in an artistic pursuit even if their origins appear largely functional.

It is telling that John Cage points to the creative possibilities of sound effects in his famous Credo:

When Cage delivered his Credo as a lecture in 1937 in Seattle, development of the creative possibilities of sound effects was, of course, in full progress in film-making. A fascination with making new sound constructs, directly supporting visuals or dramatic intention, found a ready home in suspense and then sci-fi genres of the first two decades of ‘talking film’. Functionality was initially more important than providing a sonic commentary that could illustrate the psychology of a scene. In the 1950s, new films in the science-fiction genre provided a good canvas for sound crews looking to go beyond sound as purely diegetic.

In The Day The Earth Stood Still (1951), the opening flight and subsequent landing of a UFO calls for sounds that intend to make the scene believable whilst at the same time instilling a sense of awe in the viewer. In The War of The Worlds (1953), although the music is performed by a conventional orchestra throughout, the moments when the aliens need to be characterised, synthetic or manipulated sounds are used, sometimes leading into or out of the orchestral texture. However, experimentation with sound on film was not limited to the sci-fi genre. In 1954, nature-documentary film-maker Jean Painlevé went further in his use of sound effects by creating the underwater sounds of sea urchins through recordings of ‘young people playing pots and pans’ (Hammerton Reference Hammerton2006). Painlevé called this soundtrack ‘organised noise’ and stated that it was a homage to his compatriot Edgard Varèse. Upon listening to this soundtrack it is apparent that the sounds have been processed at least in terms of editing, speeding and reversing of audio segments of the original recordings; the same sounds function at times in a diegetic manner and alternatively as straightforward musical accompaniment.

Notwithstanding the experiments of Painlevé (and sound crews within the Hollywood studio system in sci-fi and ‘B’ films), the promise of celluloid to become a medium for sound was not quite realised for composers or sound artists at this stage. The editing possibilities offered by film were ample, yet the cost and access to the tools were beyond individual artists. There were industrial film editing machines, such as the Moviola (invented by Iwan Serrurier in 1924), which were available to film-makers through the production infrastructure of movie studios but too expensive for anyone wishing to investigate sound for purely artistic purposes. By providing up to three soundtrack channels, Moviolas enabled creativity to flourish with the purpose of designing and assembling sound for images, yet other than serving as a final master format for music on film, they were largely inaccesible to composers and abstract sound artists.

Much has been discussed regarding Forbidden Planet (1956) and its ‘electronic tonalities’ by Louis and Bebe Barron, so we will not dwell on this. Although not their first incursion into using electronic music in film, it is the first important and large format instance of electronic sounds being used to complement filmic images instead of traditional music. The sounds conjured by the Barrons oscillate between literal representation and mood setting, thus the distinction between sound effects and musical accompaniment becomes blurred.

Just three years later, First Spaceship On Venus (Silent Star), by Kurt Maetzig (1959), employs the same approach to the soundtrack. Composer Andrzej Markowski creates, with synthesised means, the sound of a language which has been codified into a rock found in the Gobi desert. This rock contains a message from outer space and a team of scientists must translate the sounds into something intelligible. Synthesis is used extensively to animate the rock’s message, but this device does not stop there. During the film’s space travel sequences we have these synthetic sounds doubling as sound effects and strange musical ambience. This being a film produced behind the Iron Curtain (an East German and Polish coproduction) it is not certain how much the makers were aware of Forbidden Planet, but the sound treatments are very similar. Since there is hardly anything written on this film, one can only go by an aural impression, so I would suggest that perhaps there is more conventional synthesis in Silent Star than the circuit bending approach that the Barrons are known to have used. In both cases, though, the continuously changing interaction between the electronic sounds and the image ranges freely from apparent conformance to contest (Cook Reference Cook2001).

Also in the late 1950s, an important multimedia event took place that involved film and, in a term coined by one of its creators, ‘organised sound’: Poème électronique, an installation by architect Le Corbusier in collaboration with Iannis Xenakis and Edgard Varèse (1958) for the Brussels World Fair of 1958. Le Corbusier had conceived the Philips Pavilion as a unit of architecture, diffused sound and projected image. Le Corbusier was thinking of an immersive environment where the different media interacted with one another in space, and it is in this light that we must evaluate Varèse’s contribution. Although the composition by Varèse is a well-known piece in its own right for students of electronic music, there seems to have been little critical discussion of the fact that this piece was written for an installation involving film, multiple loudspeakers that allowed for sound placement and the architectural space itself. It was not commissioned as a piece of concert music. In this sense it may be considered sound art for film as well as being a very early multichannel installation piece. From a poietic point of view, to use the terminology of Nattiez, Varèse’s piece is evaluated today mostly as a piece of tape music yet from the point of view of the original audience, aesthesically, this music must have shown some coherence with the visuals it accompanied and the pavilion itself. Varèse’s compositional choices in his contribution to the Poème électronique event could further be perceived as a cinematic event, since the mere juxtaposition of the two media would create a counterpoint of synchronism, diegesis and mood setting. Finally, Varèse, Le Corbusier and Xenakis’ experimentation with localising sound at Brussels (400 high-range loudspeakers and 25 bass loudspeakers were used) was also seeded in film developments as far back as 1930 when, as explained by Rick Altman in Sound Space (Altman Reference Altman1992), projectionists tried switching, during the film, between an ‘orchestra’ and a dialogue speaker. Altman also discusses how the Society of Motion Picture Engineers considered in 1930 that differently placed speakers with separate sound tracks (belonging to the same multitrack) would aid the effect of sound when synchronised to picture. It seems that Poème électronique was a grand realisation of what film sound placement had aspired to and in fact would develop into towards the end of the twentieth century.

4. Beyond sound effects

The 1960s and 1970s would witness a substantial growth in the use of sound on film further liberated from its original diegetic function. Mauricio Kagel – the Argentinian-born, German composer (1931–2008) – is well known in the contemporary music world for his musical theatre pieces but also for his films. These he wrote and directed himself and, surprisingly, there has not been much writing on the subject at all – perhaps because it straddles music, electronic music and experimental film, yet resists classification into any of these fields alone. In a film such as Antithèse (1965) we have a fine electronic soundtrack, which functions much beyond the role of ‘effects’. A scientist, perhaps a studio engineer, is seen to gradually descend into madness as he tinkers with his equipment. The sound is an element here on equal footing with the visuals. Both media receive equal attention yet it is unclear, even during what seems like an electronic music interlude (a voice-over says ‘transition one’), when the surreal film plot is led by the sound and when the latter is serving the image. Antithèse is unique when viewed from a music and sound point of view, although it seems to owe something to 1950s sci-fi.

Kagel’s production in this area does not stop at Antithèse. In Match für drei Spieler (1966) two cellists on rotating chairs duel whilst being accompanied by a percussionist. This is a film of a music performance that becomes a true visual soundtrack when projected. In being a film about the soundtrack, various forms of contest and complementation (Cook Reference Cook2001) begin to take shape between the visual and the sound media, yet a strangely unified discourse is produced. Opening sounds of tennis playing alert the listener to the amusing musical bellicosity of the piece and the percussionist’s interventions function as both sound effects and sound icons, as in a military-like drum-roll section at the end of the first third of the film or in the use of the sound of dice being cast on a table. Amongst his other films, perhaps the most topical in terms of sound art is Hallelujah (1969). In this film a chaotic yet arguably exhaustive study of music and sound is realised. The film includes an explanation of the vocal production system, extended vocal techniques, sound effects and natural ambiences. It also includes scenes where loudspeakers are filmed simply projecting sound. As we are led through implausible juxtapositions of images where the sound is either produced on screen or overflows as accompaniment to the next, we have time to reflect on the role of the soundtrack, which articulates the formal structure of the film. In one scene, amongst the sounds of birds a man walks through a park with his briefcase, and the sound continues only for us to discover that the ‘bird’ sound is actually produced by a soprano in a wheelchair who ends the scene laughing hysterically. Further than a contest between sound and visuals, perhaps this sequence could be seen as an allegory of the work of the foley artist, showing that sounds which appear believable on film through the phenomenon of synchresis are often unbelievable in real life.

The fascination with filming the soundtrack as it is being created, and the film becoming partly a document of the soundtrack whilst the latter articulates a related plot-line, was also present in video at the time. Bruce Nauman, a conceptual artist who had originally been a musician, filmed Playing a Note on the Violin While I Walk around the Studio in 1967–68 (Licht Reference Licht2007). Untrained in violin playing, Nauman records himself on video playing one note on the instrument as he walks around the room. The sound is, however, out of sync with the image. Although the video could be said to somehow echo Kagel’s Duo of the same year, where guitars are played in an untrained way and in surreal scenarios, Nauman’s minimalist offering produces a further disjunction in its avoidance of synchronism. A later video, Lip Sync (1969) is finished roughly at the same time as Kagel’s epic Hallelujah. It shares the similar subject matter of vocal sound production, and Nauman presents another minimal and conceptual approach by breaking synchrony between the sound of the words being pronounced, ‘lip sync’, and the image of the lips pronouncing them (Licht Reference Licht2007).

Beyond the avant-garde, where sound liberated from its functionality is clearly at home, we still find innovative and groundbreaking sound art. In Masaki Kobayashi’s Kwaidan (which broadly translates as ‘ghost story’ from Japanese, 1965), as the opening credits roll we read: ‘Toru Takemitsu – sound effects’. Not relegated to the final list of foley workers and re-recorders, the work of the great Japanese composer is actually musique concrète composition for the screen. Kwaidan consists of four short stories, the first of which deals with a Samurai who divorces his wife in Kyoto in order to seek better fortune elsewhere. The second wife gives him social standing and fortune but he is unhappy and decides to go back to the first wife, only to find her ghost instead. The sound effects assembled by Takemitsu are not in any way traditionally deployed; they are overtly musical. Together with minimal ambience tones, percussive sounds are sequenced throughout the film, helping to tell the story in a musical way and not as mere support or augmentation to events on screen. The director goes so far as to eliminate the production sound in favour of the ‘sound effects’ whenever necessary, even if these will not synchronise with anything we see. It seems that we are presented with either Takemitsu’s sounds or the sparse sound of the action taking place. Kobayashi tries to solve the potential confusion between the roles of sounds in this way. Takemitsu provides a secondary narrative in his choice of sounds, as we gather from the twig-break sound that introduces the first story and is later used there to remember the first wife. Other sound objects include cymbals and possibly prepared piano, always with the sheen of studio processing. In a way similar to what an orchestra would do, Takemitsu’s sounds evoke the mood of what we see as opposed to providing its real or augmented sound. For instance, the sound of the first wife working her weaving machine is represented by a rhythmical construction which could credibly be the sound of a wooden machine at work, yet it does not synchronise and we are shown that this ‘sound effect’ is actually a sound-sign, a leitmotif for the first wife. This sound object, created by Takemitsu, enriches the scenes where it appears by creating an ambiguation, a second reading which is parallel to the diegetic process. It sounds like an ‘effect’ yet it works like music. It complements the reminiscences of the samurai, giving them a dreamlike character, while synchronism is carefully avoided.

A similar and contemporary approach to the sound in Kwaidan is found in another documentary by Jean Painlevé. In Amours de la pieuvre (Love Life of the Octopus, 1965), the French director collaborates with Pierre Henry from the Groupe de Recherches Musicales in Paris. Henry’s piece is eclectic and combines abstract electronic sounds and electroacoustic techniques with occasional incursions into more traditional musical ideas played on synthesisers (as when an amusing synthesised rendition of The Volga Boatmen appears in the octopus love-making scene). As in other cases discussed earlier, the role of the soundtrack oscillates between diegetic and extra-diegetic roles. Something interesting to note from the examples considered above is how much sound work that is stylistically ‘modernistic’ joins mass media through the back door provided by cinema.

In the soundtrack to Solaris (directed by Andrei Tarkovsky 1972), Eduard Artemiev reveals for us the psychic force-field that the planet Solaris exercises on the orbiting space station, as the character Kris Kelvin, a psychologist, arrives in his spaceship. Kelvin is there to investigate the failure of a mission where the team have lost contact with Earth, only to find them victims of wild hallucinations. As he first explores the seemingly empty station, he is accompanied by the eerie hum of electrical devices and atmospheric sounds that cannot be completely ascribed to any given source. The latter comes to be a constant reminder of the power of Solaris as, eventually, Kelvin also falls prey to the hallucinatory effects of the planet. In this film, we know that Tarkovsky specifically asked for no music to be composed in the traditional sense. Artemiev offered an electroacoustic score, and although in the end the soundtrack features a chorale by J.S. Bach, ‘Ich ruf’ zu dir, Herr Jesu Christ’ (BWV 639) at certain points in the film, the main soundtrack is in fact a work of sonic art which blends subtle electroacoustic sounds with the production sound of the film.

The oscillation between diegetic and non-diegetic roles can also be found in animated film work as is exemplified in Le Labyrinth (1969) by Polish film-maker Piotr Kamler with electroacoustic music by Bernard Parmegiani. Kamler collaborated also with François Bayle among others.

5. Sound Design

Initially not more than an accidental term, sound design has become a way to credit the sound artist on film. In a similar way to how the Barrons’ work had to be credited as ‘electronic tonalities’ on Forbidden Planet, Francis Ford Coppola explains how the sound designer title arose. In discussing the difficulties Walter Murch had to overcome to finalise and mix the sound for The Rain People (1969), Coppola tell us:

Having been named, it seems, sound design legitimised and encouraged a re-evaluation of the role of the sound artist on film. Having given Murch his role, Coppola would then ask him, since he considered this film would be in effect a ‘sound composition’ (Ondaatje 2002: 53), to collaborate on The Conversation (1974). Alan Licht believes that this film ‘resonates with the use of repetition in the work of Naumann, La Monte Young, Philip Glass, and Steve Reich of the same period’ (Licht Reference Licht2007: 208) and it is not too difficult to agree with him. Once the work of the sound artist is elevated and ranked similarly to music, it is explicitly allowed a non-diegetic role, one that shadows its more functional role of representing the listening experience for the viewer.

Freely using recordings of everyday sounds to complement the visual world of cinema can have either a multiplying semantic effect or quite the opposite. When detective Harry Caul, in The Conversation, clearly understands the speech he has eavesdropped upon and registered on tape, our mind moves along to the next puzzle; that one is solved. Yet when he replays a segment endlessly where background noise obscures the distinction between speech and other sounds, the possibilities for meaning seem infinite. The way Coppola conceived of the film plot and Murch worked on the detailed sound of recording, playing and scrubbing through Caul’s tapes can thus easily be accepted as a composition in sound.

This preoccupation with the storytelling potential of designed sound can also be observed in the video-art world as in Bill Viola’s A Non Dairy Creamer (1975) where the sound is purposefully detailed (Licht Reference Licht2007). In fact, Viola, who has ample experience of sound and at least once collaborated with David Tudor in the preparation and setup of Rainforest IV, frequently utilises manipulated sound as both diegetic sound and incidental music. In Five Angels for the New Millennium (2001), five large cinema screens are arranged in a sizable darkened room. Divers are seen exploding out of the water in very slow motion (the films are upside down). The sound, the recording of a water dive, is played as such a slow buildup that although we are able to associate it with water we may equally perceive it as an eerie musical gesture that conveys the vastness of the coloured water-space which slowly splashes on screen. According to Rhys Davies in The Frequency of Existence (2004), the sounds in Five Angels are not simply designed as the slowing down of the water splash, but water elements (drops, bubbles) are added further to help enhance the visual narrative of motion. In one of the Angels, where the camera is situated above the water, the sounds of air and night insects help place the viewer within a drone of ‘continuously evolving, unbroken sound’ which Davies terms ‘archetypal sound’; the sound of key natural events such as thunder, wind, lightning, ‘the roar of a predator’ or ‘the cry of a newborn’ (Davies Reference Davies2004: 148). For him, these archetypal sounds convey ‘the frequency of existence’. Interestingly, when discussing the use of sound in the work of Viola, Davies draws parallels with the work of Ben Burtt in George Lucas’ Star Wars: A New Hope (1977). He singles out the sound of the first imperial spaceship we see and perceives it as being reminiscent of archetypal sound, and those drones much loved by Viola. Of course, his discussion could equally apply to the work of Murch in THX1138, and later work such as The Conversation and Apocalypse Now (1979). Murch also went beyond manipulating sound to manipulating the recorded music film-score, as can be heard in his editing and relayering of Nino Rota’s music for the climactic severed horse-head scene in The Godfather (1972) (Licht Reference Licht2007). From the interviews with Murch featured as a bonus to the THX1138 DVD released by George Lucas in 2004, we also know that a great part of the music of the film was inspired in Murch’s tape music manipulations of the Pergolesi Stabat Mater (1736). He also explains in great details many of his musique concrète techniques for creating the sound of futuristic cars (edited tape recordings of a jet-plane fly-by) and motorcycles (processed sounds of women screaming, recorded inside a bathroom).

The development of electronic music instruments in the 1980s allowed sound design to flourish in its newly recognised role within film-making. The democratisation of editing and production tools has enabled any interested person to try their hand at creating sound for film or video. In fact, the distinction between working for film or video has been blurred for sound artists. In the past, specific knowledge about film footage and its relationship to tempi was necessary to create sound or music for film. Intricate calculations needed to take place so as to work out how to ‘hit’ a particular cue with a sound or a musical phrase. The process of editing, realised on ‘flatbeds’ – table Moviola machines – was manual and there had to be a close cooperation between editor and creator. The age of personal computers has gradually done away with many previous technical limitations and brought interesting developments to sound artistry on film.

6. Sound design and music

Arguably, for new generations who have come to audiovisual work through computers and digital video, there is no great distinction between different formats. Now, SMPTE frame offsets can be easily determined by clicking and dragging on a graphic user interface. Since the 1990s increasingly more composers for media have also promoted themselves as sound designers, resulting sometimes in music that contains elements of sound work, and sound work that contains elements of musical thinking.

Aesthetically, perhaps we are at a point where musicians and sound designers inhabit the same creative space. Yet the ease with which pre-recorded musics can be processed, edited and layered to image, and mixed with sound effects, foley and ambience recordings has arguably made music an element of a larger sound canvas within film. Rather than proclaiming, as Fischinger, Cage and Varèse did, that all sounds are or can be music, the new school of thought should be that music is another component of sound composition. Young sound artists working for visual media are equally at ease nowadays with slicing music loops as layering the sounds of explosions onto a soundscape, all the while syncing to picture.

In films such as Delicatessen (dir. Marc Caro, 1991) sound becomes an omniscient narrator as it travels through the ventilation shafts of the Parisian apartment building where the characters live. We are guided through the plot as the sound of conversations, TVs and music practising are dispersed through the building. It overtly competes with the music to take on an emotion-colouring role, whilst the latter is often reduced to a diegetic one. In a scene where two of the characters are tuning the ‘moo’ sounds of toy-cows, an effective musical sequence is produced out of the rhythmic testing of the pitch of the toys, thus blurring the distinction between diegetic and incidental music roles. In recent films such as A Scanner Darkly (dir. Richard Linklater 2006), composer Graham Reynolds processes his acoustic instruments through digital signal processing plugins so thoroughly (Hurwitz Reference Hurwitz2006) as to blur the distinction between foley effects and music. The opening scene, for instance, where imaginary bugs sprawl out of the character’s hair seem like a cross between acoustic contrabass plucking in the high register and sound effects for ‘insect sounds’. It all works musically, yet the music occupies a particular space as an audio element within the overall soundtrack. In the final production, this is controlled not by a musican but by a sound editor (should read ‘artist’); imposing a more overarching set of criteria upon the musical contribution as part of the whole soundtrack.

7. Sound in live Cinema

The development of greater computer interactivity with digital media has also brought a fresh approach to cinematic sound and image. It is now easier than ever to program the image to react to sound and vice versa, as well as modulating both through sensors or external hardware. Languages such as Processing (a graphics programming language by Ben Reas and Casey Fry), vvvv (developed by Joreg, Max Wolf, Sebastian Gregor and Sebastian Oschatz) or Jitter (the graphic and visual manipulation toolkit for software developer cycling74’s MaxMSP programming environment) as well as custom-made applications are easily accessible to any interested party. Mathematical models and all kinds of data can be translated into image and sound. Abstraction in cinematic creation has arguably been given a new lease of life by collaborations between electronic musicians and visual artists. The medium is no longer the 35 mm celluloid or the 35 mm magnetic tape. Frames are no longer divided into the four sprocket subdivisions, but cut down to the thousandth fraction. A growing number of digital artists are willing still to call this practice ‘cinema’, dampening the ancient memory of playing live music to silent film by creating film out of sound and music interacting with often abstract image.

The idea of cinema has thus been extended by a confabulation of electronic musicians and VJs turned film-makers. In a quest to go beyond the ambience-setting task of a VJ or the straight mapping between visuals and sound that can take place in light shows, live cinema artists are trying to develop a new approach to abstract film. They aim to make their work the focus of a performance event: a screen, the audience in darkness and immersive sound. To quote one of them, Boris Debackere, ‘Cinema is a virtual capsule with a projection screen as a window, which gives access to an audiovisual trip through time and space’ (Debackere Reference Debackere2008). Artists like Klipp Av (Collins and Oloffson), Frank Brechtschneider, XNOGRAFIKZ (Arturo Gil and Martha Cervantes), ROTOR (Boris and Brecht Debackere) amongst many others are all engaged in what can be termed as a live cinematic exploration, yet the way they will categorise themselves may vary substantially. What they all have in common is their use of live computer interaction between sound and images that are presented in a large-screen format during performance or installation. This work can be perceived as cinematic and evaluated with reference to the broader referential space of film. From cinema they inherit key aspects of the audiovisual language regarding issues of synchronism, the perception of image versus sound size, editing rhythms and so on. What interests us is that their exploration is largely informed by sound and its manipulation processes on one hand, and the exploration of sound and image together in an improvised or generative setting on the other.

In Boris Debackere’s Probe (2008), generative sound and image are triggered by the viewer as they move within the projection theatre, allowing for an immersive interactive experience. There is no doubt that Debackere’s concerns regarding the relationship of sound to image is grounded in the culture of film. His is an art which directly descends from the practice of the, now traditional, sound designer for film.