Sound -> Object -> Gesture: Physical Affordances of Virtual Materials
Jon Pigrem, Andrew McPherson, Nick Bryan–Kinns, Robert H. Jack · 2022
This paper explores the influence of materials communicated through the sound of digital musical instruments. We are particularly interested in how musicians approach the design of new instruments when the sounds they make are constrained to recognisable material timbres and behaviours. We present a digital musical instrument (DMI) design study in which 20 participants design 80 new DMIs using an instrument kit. We use enactive approaches to design to enable the participants to borrow from everyday understandings of interaction in the real world, revealing aspects of their environment mediated purely through sound during the study tasks. We demonstrate that the influence of virtual material is strong, with participants not only taking material cues from the sound, but in places interaction strategies and performance approaches. We suggest that in addition to commonly referenced modalities of sight and touch, sound can be both precursor and mediator to our selection of, and gestural interaction with, materials at the design stage. We present two specific approaches to instrument design (Sound, Object Gesture – SOG, and Sound Gesture Object - SGO) which we found to underpin participant’s approaches to the design task and have wider effects on the instruments they designed, and the model of creativity employed in their use.