1. INTRODUCTION: THE NOTATION OF NON-LINEAR MUSICAL FORMS

Since the early 2000s, a growing number of composers have developed performance scores which are generated in real time and displayed for performers on laptop screens or tablet computers, or via video projection systems (Smith Reference Smith2013a). Common to many of these scores is the adoption of radical notational schemas designed to facilitate the development and exploration of extended non-linear musical forms and processes.

While non-linear processes can manifest themselves in many levels of a musical work, it is arguably their use in large-scale structural organisation that has received the most attention. At their most basic level, such processes call for performers to determine the sequence of discrete musical sections of a work during performance. In Earle Brown’s Available Forms I (1961), for example, the conductor determines the order of the work’s various subsections, and indicates to the ensemble the next subsection to be performed through various hand gestures. Similarly, in Stockhausen’s celebrated Klavierstücke XI (1957), it is left to the pianist to determine the order of the work’s nineteen discrete musical sections all of which are arranged around a large, single page score; see Figure 1.

Figure 1 Excerpt from the score for Karlheinz Stockhausen’s Klavierstücke XI (1957), in which the pianist determines the ordering of nineteen musical fragments. © 1957 Universal Edition (London) Ltd London/UE 12654. Used by permission of Hal Leonard Australia Pty Ltd ABN 13 085 333 713. All rights reserved. Unauthorised reproduction is illegal.

The ability to implement more complex musical orderings and circumvent decision-making biases is one of the many features that has attracted composers to the use of scores generated in real time. In the author’s Valses and Etudes (2005, rev. 2010), for piano and computer, for example, a score is generated by computer and displayed for the pianist from a collection of score fragments of various works for solo piano which succeed one another according to a first-order Markov chain procedure (Kim-Boyle Reference Kim-Boyle2010). This procedure uses weighted probabilities to determine the likelihood that one score fragment will follow another. There may, for example, be a 30 per cent probability that score fragment A will follow score fragment B, a 0 per cent probability that score fragment B will follow score fragment B, a 60 per cent probability that score fragment C will follow score fragment B, and a 10 per cent probability that fragment D will follow fragment B. Without automating this selection process, it is unlikely that a pianist would be able to implement such a desired probability weighting and score ordering during performance.

The blossoming of interest in non-linear musical forms in the 1950s and 1960s was preceded by a greater interest in non-narrative structures in various other art forms, including experimental film (Rees Reference Rees2011), and perhaps more notably in literary genres (Eco Reference Eco1989). This ran alongside a growing awareness of the relationship between narrative structure and typographical presentation most strikingly typified, perhaps, in the poetic experiments of Stéphane Mallarmé of the previous century. The development of non-linear narrative forms rarely, however, resulted in such a radical break with traditional notational schemas as profoundly as they did in music. The concern with visual presentation and design typified in the scattered fragments of notation for Stockhausen’s Klavierstücke XI were reflected and amplified in a growing body of work exploring non-linear forms such as the mobile-forms of Haubenstock-Ramati’s Liaisons (1958) or Mobile for Shakespeare (1960) (Haubenstock-Ramati Reference Haubenstock-Ramati1965), or the open forms of Christian Wolff’s Edges (1968) or For 1, 2 or 3 People (1964) (Wolff Reference Wolff1987), where the traditional linearity of the notated page is subverted through graphically notated pages that establish environments for musical play; see Figure 2 for example.

Figure 2 Excerpt from the score for Wolff’s For 1, 2 or 3 People (1964).

In some cases, the exploration of non-linear forms prompted the fixity of the score itself to be challenged, with performers required to create an instantiation prior to performance from a set of pre-notated elements provided by the composer. In Yasunao Tone’s Anagram for Strings (1962), for example, performers create a performance score by drawing lines, interpreted as glissandi, on a template randomly filled with various black and white circles each of which indicate glissandi durations (see Figure 3), while in John Cage’s Fontana Mix (1958), a score is created from the superimposition of a collection of pre-inscribed graphic transparencies. While the graphic typography in these scores is generally consistent from one version to another, yielding a degree of consistency and uniformity of sonic utterance from one realisation to another, the micro-level distribution of musical events naturally differs from performance to performance. The complex preparations often required in creating scores such as these can easily be included in live generative processes of real-time scores, see for example the Australian new music ensemble Decibel’s real-time realisations of the scores for Cage’s Variations I, II, and III (Vickery, Hope and James Reference Vickery, Hope and James2012), and the non-linear distribution of musical events at the micro level is also often explored in the work of composers developing scores which are generated in real time.

Figure 3 Template for Tone’s Anagram for Strings (1962) upon which glissandi lines are drawn by performers.

As is evident, many of the aesthetic concerns and musical challenges facing today’s composers developing works with real-time screen scores are grounded in this body of work of an earlier generation of composers. While such scores are also well suited to the exploration of non-linear forms and processes, the ability to vary the way in which notational information unfolds over time, however, represents a radical extension. This has, in turn, brought a renewed awareness to the importance of score design and graphic typography and the way both mediate musical performance.

2. SCREEN-SCORE DESIGN

While a score’s visual design has been of particular interest for a large body of contemporary composers (Cage and Knowles Reference Cage and Knowles1969; Sauer Reference Sauer2009), such a concern is by no means a contemporary phenomenon. One only need recall the work of Baude Cordier (Figure 4) and other composers from the Ars Subtilior movement of the fourteenth century whose strikingly beautiful manuscripts help underscore more formal musical properties of the works they represent.

Figure 4 Excerpt from the score of Cordier’s Tout par compas suy composés (c. 1400).

The development of real-time screen scores, however, introduces a series of design considerations for composers that do not ordinarily pertain to paper-based scores. While the materiality and constraints of the printed page may no longer be delimiting factors, the typically lower resolution of a screen-score tends towards a greater reliance on graphic typographies, while the relatively small display space of a screen is countered by an increased reliance on animation techniques. And even though the type of animation techniques employed may vary widely, they can, following McClelland and Alcorn, be broadly categorised into three types of display modes – pages, scattering and scrolling (McClelland and Alcorn Reference McClelland and Alcorn2008), each of which frames visual data and its representation of non-linear processes in distinctive ways.

In a pages display mode, the visual information displayed on the screen is systematically replaced by new data one page at a time. This framework is well suited to simple open-form structures with interchangeable musical subsections such as the author’s Valses and Etudes. The temporal dynamics of scrolling and scattering display modes present more complex design issues, which will be better appreciated through a more detailed examination of a select number of representative screen scores.

In a work such as Lindsay Vickery’s Ubahn [sic] (2012), for two violas, two cellos, double bass, percussion and electronics, for example, performers read their individual part from a networked tablet computer display which presents fragments of common practice notation overlaid on a circa 1985 map of the Berlin U-Bahn; see Figure 5.

Figure 5 Snapshot from the score for Vickery’s Ubahn (2012).

Each of the instrumentalists commences performance of their distinctly coloured line from a predetermined station. As they perform the musical material inscribed along their line, the score scrolls and they arrive at other stations before their line is directed to another station randomly selected by the computer. After six minutes, the computer selects the shortest path to an ‘East German’ sector, coordinating arrival of all six performers at the final section of the piece. The scrolling speed of Ubahn’s individual lines is clearly constrained by the requirement of performers to be able to distinguish the notation inscribed along them (Vickery Reference Vickery2014). To further facilitate legibility, the map is also rotated as performers read their lines. These various design considerations manifestly affect the rhythmic characteristics of the work with a notable absence of rapidly articulated passages or passages that require tight synchronisation between individual lines.

Similar design issues permeate the screen scores for Cat Hope’s Longing (2011) for voice, bass clarinet, viola, cello and percussion, and Miss Fortune X (2012) for a.m. radio, cello, viola, piano and percussion, although both scores embrace a more overtly graphic design aesthetic. In these works, new musical information is progressively unfolded for each performer on a series of networked tablet displays (see Figure 6). The scores developed for both works utilise a fairly simple graphic typography that nevertheless incorporates some traditionally notated symbols for dynamics and various other articulations. The notation is based on a proportional pitch–time grid with changes along the y-axis interpreted as changes in pitch contour, and articulations of lines on the x-axis corresponding to changes of phrase duration. Clearly, as in Ubahn, the speed at which the scores scroll is constrained by the necessity to maintain legibility, which is especially needed for reading traditionally notated articulations and dynamic indications.

Figure 6 a) Score snapshot from Longing (2011) (left); b) Score snapshot from Miss Fortune X (2012) (right).

The design of both Vickery’s and Hope’s scores is also conditioned by various sightreading constraints (Vickery Reference Vickery2014). For example, the range of colour employed is deliberately constrained by the ease with which individual colours can be distinguished on a screen. Similarly, there is also a tendency to use thick lines to indicate important directives such as pitch contours rather than finely articulated graphic indications which, as noted, are clearly manifested in the temporal dynamics of the works. The constraints of sightreading and screen resolution combined thus tend to limit the amount of information displayed on the screen at any one time.

As noted by McClelland and Alcorn, the amount of peripheral information on a screen may also be reduced in order to maintain visual focal points (McClelland and Alcorn Reference McClelland and Alcorn2008). Such scattering display modes (McClelland and Alcorn Reference McClelland and Alcorn2008) are employed in the author’s point studies no. 2 and line studies no. 1. In line studies no. 1 (2013) for piano and computer, the score is generated in real time during performance and consists of a set of concentric rings subdivided into arcs of varying lengths and thickness; see Figure 7a. Each arc denotes a pitch to be played, with arc lengths corresponding to durations, and thicknesses to dynamics. As the score develops, arcs are progressively extended, overwritten and erased. The rotation of the arcs changes their alignment and affects the way in which the pianist is able to navigate through the score. Aside from the animated elements of the score, the underlying concept is similar to that employed in Earle Brown’s Four Systems (1954); see Figure 7b, in which the length and thickness of lines correspond to variations in duration and dynamics respectively. With the use of dynamic generative processes, however, the work introduces new musical possibilities from performance to performance and with its circular, continually overwritten layout helps to transcend the linear biases of interpretation inherent in the score for a work such as Brown’s.

Figure 7 a) Score snapshot from Kim-Boyle’s line studies no. 1 (2013) (upper); b) Score excerpt from Brown’s Four Systems (1954) (lower). © 1953 Associated Music Publishers Inc. G. Schirmer Australia, Pty Ltd. International copyright secured. All rights reserved. Used with permission.

In the author’s point studies no. 2 (2012) for any two pitched instruments and computer, different pathways through the works pitch structure become possible through the three dimensional positioning of coloured nodes. In the score for this work, pitches are represented by connected, coloured nodes in a rotating three-dimensional grid; see Figure 8. Performers navigate through the score by following connections between the nodes. The colour of the node denotes the pitch to be performed with the size of the node indicating its dynamic. The length of the connections between nodes indicates duration. Nodes are progressively removed from the score during the course of the piece, which necessarily limits the pathways along which performers may navigate. The sounds of the performers are processed through a MaxMSP patch such that the position of the nodes in 3D space affects the live spatial processing of the performers as well as the spatial processing applied to an accompanying computer-generated audio interpretation of the same score. The deliberately constrained graphic typography also facilitates interpretation. There are, for example, only five easily distinguished colours employed (white, red, yellow, blue, green) with a greater distribution of whites towards the centre of the grid.

Figure 8 Screen snapshot of the score for Kim-Boyle’s point studies no. 2 (2012).

Through extending the traditional up–down, left–right navigation across a two-dimensional surface, 3D notations allow additional layers of musical information to be efficiently integrated within a notational schema (Reyes-García Reference Reyes-García2013). For open-form and other works based on non-linear forms, this provides additional possibilities for the way in which such notations can be transformed. This can include, for example, making the notation responsive to live performance, the real-time directions of the composer (McClelland and Alcorn Reference McClelland and Alcorn2008) or even the audience (Freeman Reference Freeman2008). In Pedro Rebelo’s Netgraph (2010) performers situated in various networked performance sites directly influence the development of a three-dimensional graphic score through their musical interpretation of that same score. Each of the three performance sites is presented with different camera perspectives of the dynamic three-dimensional score and extend it with uniquely coloured graphic objects (Rebelo Reference Rebelo2013). Through subtle variations of sound quality and more direct interaction with control interfaces, performers are able to affect various aspects of the notation including drawing speed and various properties of the object’s trajectories (Rebelo Reference Rebelo2010). A screen snapshot of the graphic notation from Netgraph is shown in Figure 9, with a snapshot taken from the work’s premiere directly below.

Figure 9 a) Score snapshots from Rebelo’s Netgraph (2010) (upper); b) Photo from the premiere performance (lower).

As in the author’s line studies no. 1, the dynamic unfolding of Netgraph’s score occurs within a display framework in which peripheral visual information is reduced with a corresponding focus drawn towards the centre of each of the display screens. The use of a three-dimensional graphical representation, however, enriches the visual score and interpretive possibilities for each of the performers.

Adopting Pold’s analysis of gaming frameworks (Pold Reference Pold2005; see also Salen and Zimmerman Reference Salen and Zimmerman2003), the metaphor of navigation has considerable value in the evaluation of a screen-score’s visual design. And following Galloway’s recognition of the importance of play in systemic interaction (Galloway Reference Galloway2012), the real-time score can be considered as defining a dynamic environment that performers explore through musical gestures with the constraints of this freedom clearly demarcated by the notational schema. Whether these constraints become material for compositional or performative investigation naturally depends on the particular work. In the author’s line studies no. 1, the path that the performer chooses to navigate, or use to playfully explore their way through the circular score, is left to their discretion, but it is constrained by the alignment of rotating arcs. It is this temporal dynamism inherent in the score itself and how this conditions the systemic interaction of the performer that is, of course, the critical difference when compared to works in fixed media such as Brown’s Four Systems or Wolff’s For 1, 2 or 3 People. Indeed, the manner in which temporal dynamics are integrated within the formal structure of a real-time score – whether through scrolling paradigms, responsiveness to performance gesture or basic graphic visualisation of non-linear musical processes – becomes a defining quality of the work.

3. SCORE MEDIATION AND TRANSPARENCY

While screen-scores have been shown to be especially well suited to the representation of non-linear processes and other non-linear musical forms, their distinctive design characteristics strongly mediate the ways in which performers engage with the musical processes they prescribe. This is also true of course of fixed-media graphic notation, but the mediation is foregrounded to a greater extent through the tendency to project such scores for audience viewing, focusing particular attention on the relationship between graphic typography and visual design, and sonic utterance. To the extent that screen-scores represent sonic data and data structures, they share many of the defining characteristics of digital interfaces (Pold Reference Pold2005), and to that end their transparency within a performance context presents similar concerns to those faced by interface designers. In this respect, the emerging field of interface aesthetics provides a useful framework for understanding some of the key issues (Galloway Reference Galloway2012).

In his 2005 paper, Pold examines how the foregrounding of the interface has become a critical area of enquiry for the digital arts (Pold Reference Pold2005). Through providing various degrees of functional realism, artists are able to make the mediatory power of the interface a locus of aesthetic interest in its own right. While the extent to which these concerns apply to real-time scores varies from work to work, given the pervasive tendency for performances of such works to project the score for the audience (Hope and Vickery Reference Hope and Vickery2011), it would certainly seem to be the case that the score is foregrounded to a far greater extent than in most other types of contemporary music practice, with the possible exception of live coding performances.

Projection of the score, even if motivated by a desire to share an often painstakingly developed visual design, draws overt attention to the manner in which the performers engage with the musical processes they prescribe. Ryan Ross-Smith’s series of studies, programmed in the openFrameworks environment (Smith Reference Smith2013b), which often use very simple procedures to create complex rhythmic patterns and textures, typify this tendency. For example, in his Study No. 8 (2012) for fifteen percussionists, a series of rotating arcs, each uniquely assigned to one of the percussionists, oscillates between two fixed boundary points (see Figure 10a). When a boundary is reached, the assigned percussionist strikes the instrument on either their left- or their right-hand side. The colour of the boundary point (grey, brown or blue) corresponds to the type of instrument struck: unpitched metal, wood or pitched metallic percussion instrument respectively. The work has a relatively simple form: the arcs gradually accelerate at different rates with the increasingly asynchronous movement generating textures of complex rhythmic phasing, before being sustained at a constant pulse and gradually decelerating towards the work’s conclusion. The process is extended in Study No. 22 (2013) for twenty-four percussionists, where arc movement is synchronised across various groups, rather than modulating independently. Unlike Study No. 8, in which the score is projected on a large screen for all performers and the audience to view, in Study No. 22 the score is projected on the floor (Smith Reference Smith2014) and encircled by the ensemble. This not only embodies the score within the ensemble itself, but foregrounds the notational schema to an extent that score itself becomes a member of the ensemble.

Figure 10 a) Score snapshot from Study No. 8 (2012) (left); b) Score snapshot from Study No. 22 (2013) (right).

While the relationship between score and sonic result is easily decoded in the two Ross-Smith studies as a result of the way the composer has chosen to map graphic typography to gesture, the connection is far more opaque in a work such as the author’s point studies no. 4 (2014) for voice and computer. Written for the Dutch sound poet Jaap Blonk, the score for point studies no. 4 is generated in real time and consists of five sets of concentric arcs each articulated with nodes of various colours and sizes, and radials terminated by nodes of uniform colour and size. A unique vowel appears beneath each set; see Figure 11.

Figure 11 Score snapshot from Kim-Boyle’s point studies no. 4 (2014) for voice and computer.

The colours of the nodes along the arcs, which represent various consonants (fricatives, plosives, affricates, nasals, approximants) to be enunciated by the vocalist, are randomly assigned when the score is generated. The colours of the nodes at the terminal ends of each radial represent a distinct vowel. The size of each node corresponds to a dynamic level. During the work each arc and radial successively rotates, either clockwise or counterclockwise, to a randomly determined position, bringing different nodes into alignment, which changes the sonic possibilities each set of arcs represents and, correspondingly, the pathways that the vocalist has available to navigate through the arc sets. A computer-generated interpretation of the score accompanies the live vocalist and is drawn from a library of various pre-recorded samples of vowels and consonants.

The form of point studies no. 4 emerges as the interplay between the dynamic processes visualised through the score and the manner in which the vocalist chooses to articulate them through sonic utterance. While the score, again, takes a foreground role in defining this interplay, the way it codifies the sonic language of the work is not easily decoded by the audience. The notational/gestural transparency of screen scores thus charts a wide spectrum from the easily decoded relationships of Study No. 8 and Study No. 22 through to the relative opaqueness of point studies no. 4. Not surprisingly, this also charts a spectrum between scores with visual information of minimal dimensionality through scores with visual information of maximal dimensionality. Yet even in scores that can be relatively easily decoded, the tendency to project them for viewing by the audience draws specific attention to the notational schemas they present.

4. CONCLUSIONS

The radically new notational schemas developed for the representation of non-linear musical forms and processes, a sample of which have been examined in this paper, build on the work of artists and composers first exploring non-linear musical forms in the mid-twentieth century. The ability to integrate temporal dynamics within the representational form of a score, however, marks a distinctive innovation. Whether this use takes the form of simple animation techniques of graphic primitives, or involves complex coordination of multiple scores across networked devices, it is clear that the development of screen scores requires deeper consideration of a wide range of visual design issues which mediate the ways in which performers ultimately engage with musical processes. This enquiry comes through an unprecedented foregrounding of the score that draws overt attention to the ways in which performance is mediated by a notational schema that prescribes a field of enquiry rather than denoting a fixed and predetermined entity.