Designers of digital musical instruments often highlight the flexibility of these instruments in comparison to their acoustic counterparts. Freed from mechanical constraints, digital instruments can produce new types of sound and offer the performer new interaction modalities. Recent years have seen a proliferation of design ideas shared in both academic venues and commercial crowdfunding campaigns (Jensenius and Lyons Reference Jensenius and Lyons2017; McPherson et al. Reference McPherson, Morreale, Harrison, Holland, Mudd, Wilkie-McKenna, McPherson and Wanderley2019). As the availability of low-cost, high-performance computing continues to increase, we are rapidly approaching a point where the uptake of new instruments is limited not by technical factors but by human capabilities and cultural considerations. Perhaps as a result, the vast majority of new digital musical instruments have a short lifespan, used only by a few people before being set aside in favour of even newer technology.
As an instrument designer with a background in music composition and electronic engineering, I am interested in creating instruments to fit specific existing performance practices. Specifically, I seek to maximise the creative novelty of a new instrument while minimising the amount of re-learning needed for a performer to play it to a high standard. A core part of my task involves designing these instruments to fit human sensorimotor abilities which are the result of years of instrumental practice. However, traditional instruments also support a broader cultural ecosystem, including established repertoire, well-known virtuosi, pedagogical practices and audience familiarity. Building on these cultural resources may help increase the longevity of a new instrument.
One approach to creating new instruments that connect to existing performance practice is the creation of augmented instruments: familiar musical instruments whose capabilities have been extended with new sensors and modes of sound production. My first augmented instrument project was the magnetic resonator piano, an augmentation of the acoustic grand piano which uses electromagnets to induce vibrations in the strings, allowing the player to produce organ-like sustain, harmonics, pitch bends and new timbres (McPherson and Kim Reference McPherson and Kim2012). Since its creation in 2009, the instrument has acquired a repertoire of around twenty-five new compositions and a growing community of pianists proficient in its techniques.
Playing the magnetic resonator piano is different in important ways from traditional piano performance: for example, new techniques involve pressing the keys slowly enough that the hammers do not strike the strings or holding the keys partway down. Learning these actions can take time and practice. Therefore, for my second augmented instrument project, TouchKeys, I focused specifically on fitting the behaviour of the instrument to the constraints of existing keyboard technique (McPherson et al. Reference McPherson, Gierakowski and Stark2013). TouchKeys is an augmentation of the keyboard that adds capacitive touch sensing to each key. The TouchKeys sensors attach to the surface of any keyboard, measuring the location of the fingers in two axes: across the bass-to-treble axis (X) and along the longer axis of each key (Y). TouchKeys lets the performer add techniques like vibrato, pitch bends and timbre changes to each note without cumbersome wheels and pedals, simply by moving the fingers on the key surfaces.
I first developed TouchKeys in 2010, at a time when interest was growing in the industry in ‘multidimensional polyphonic expression’: giving the performer independent control over the pitch, dynamics and timbre of multiple simultaneous notes. Other commercial products with this capability include the ROLI Seaboard, Roger Linn’s LinnStrument, the Madrona Labs Soundplane and the Haken Continuum Fingerboard. These instruments, like TouchKeys, are controllers for arbitrary digital sounds which use the MIDI (Musical Instrument Digital Interface) or OSC (Open Sound Control) protocols. Each instrument provides some degree of user-adjustable mappings from action to sound.
These other instruments replace the traditional keyboard action with novel interactive surfaces. For example, both the Seaboard and Continuum use a continuous deformable membrane instead of discrete keys, and the LinnStrument and Soundplane choose a rectangular layout over the familiar pattern of white and black keys. By contrast, TouchKeys attempts to retain as much as possible of the keyboard technique that performers spend many years learning, even if this circumscribes the design possibilities. By designing it to be installed on top of an existing keyboard, TouchKeys preserves the layout and action of the keyboard. Due to manufacturing constraints, the touch sensors modestly alter the surface texture, and the sensor edges are less rounded than a traditional key top, but these are small changes in comparison to other new controllers.
In creating TouchKeys, I discovered that how the software behaves can make or break the player’s familiarity with the instrument. In a sense, adding new techniques is the easiest part of the design; the harder challenge is to avoid having those new techniques interfere with existing ones. One straightforward example is the use of Y-axis sensing (i.e. the long axis of the key). To assign the raw Y position to a control like pitch would render the keyboard unplayable. Redundancy in where the fingers contact the keys is a core part of keyboard technique, as it accommodates the different lengths of each finger and the different hand orientations needed to reach different combinations of keys. It would be futile to demand that the player press each key in the exact right position in order to produce the expected pitch.
The obvious solution is to design the interface so that the player can use relative finger motion after note onset, rather than absolute position, to control pitch or other effects. However, even this approach creates subtle problems, as the fingers naturally move on the keys in traditional playing. A study with 8 pianists showed that in 26 per cent of key presses, the finger moved more than 5 per cent of the length of the key, and in 11 per cent of key presses it moved more than 10 per cent of the key length (McPherson et al. Reference McPherson, Gierakowski and Stark2013). In the X (horizontal) axis, the figures are even more striking: 75 per cent of key presses had finger motion more than 5 per cent of the key width, while 55 per cent had motion more than 10 per cent of the width. The difference between X and Y probably stems from a combination of the dimensions of the keys, which are longer than they are wide, and the fact that the hands move most in the horizontal direction in traditional playing.
In order to account for the movement of the fingers along the keys in traditional keyboard technique, my solution for pitch bending is to use relative finger motion in the Y-axis with a user-adjustable threshold below which small movements are ignored. For vibrato, the software looks for relative finger motion in both directions within a certain time period before starting to change the pitch, as my studies showed that natural movements on a key in traditional playing were almost all in a single direction only. Generally speaking, the higher the threshold, the more effort is needed to engage the new techniques, but the less likely the techniques will be to engage unintentionally. It may be that as a player’s technique becomes more attuned to TouchKeys, the player can gradually reduce the thresholds to achieve greater sensitivity to small movements.
In July 2013, after three years of development, I launched TouchKeys for public sale in a Kickstarter crowdfunding campaign. The campaign raised £ 46,000 to build and ship TouchKeys kits and instruments to musicians around the world. Most of the backers on Kickstarter opted for a self-install sensor kit that they could attach to their own keyboard, though pre-built keyboards were also available.
Though I have not undertaken a systematic survey of the musical backgrounds of the TouchKeys players, my impression is that the largest player community consists of amateur or semi-professional keyboardists who maintain home electronic music studios, and who most usually play in a pop/rock style. Other players include jazz keyboardists, professional studio composers and experimental musicians. A 2014 review in Sound on Sound magazine reflects the feedback I often get from performers: ‘The nice thing about TouchKeys is that it builds upon existing keyboard techniques – you’re not learning from scratch, just adding new movements and actions to enhance the performance – and that means the learning curve is fairly shallow’ (Robjohns Reference Robjohns2014).
Musicians don’t take up new instruments solely because they are partly familiar; rather, the instrument’s creative novelty provides the impetus. An open question regarding all the novel MIDI controllers on the market is whether novelty of technique or novelty of sound is more important. In principle, the same MIDI performance could be achieved using any of the aforementioned controllers, or given more time, using a basic keyboard and MIDI editing software. Viewed from this perspective, it might not be possible to tell what instrument a performer played solely by listening to the audio output. In practice, however, each controller is going to make certain patterns and techniques more or less idiomatic, and this will end up influencing what the performer does in characteristic ways. In other words: perhaps it is possible for each MIDI controller to have its own signature sound, even when they all control the same synths.
There is no single answer for how best to balance novelty and familiarity in digital musical instrument design, and much depends on the artistic priorities of the designer and any other musical collaborators. Going forward, I hope to discover more general principles in how to design for a performer’s existing sensorimotor skills and cultural expectations.
