Interactive instrumental performance and gesture sonification
KA Beilharz · OPUS - Open Publications of UTS Scholars (University of Technology Sydney) · 2008
This article describes a system for interactive performance that generates live musical accompaniment to an instrumental performer, using a Neural Network model and granular synthesis controlled by the gestures and breath of the performer. More broadly, the technologies for gesture-tracking can be applied to various interactive environments for real-time music augmentation. The use of pitch-tracking and musical parameters derived from the performer's output seed a generative computing model that procreates new material and perpetuates musical fragments over time to produce a contextual yet create response to the live performer. Electronic music and live visualisation of gesture augment the scope of an analogue traditional instrument, in this case a bamboo Japanese flute, shakuhachi, hence the title of the interactive music environment, HyperShaku. The design issues of relevance to inter-domain implementation include: integration of A.I. modules (Artificial Life, biologically-inspired and Evolutionary processes) in a system for real-time computational data processing; using A.l. modules for sonifying data for automated and interactive generation of sound: and considerations for mapping gesture and other data to auditory display. The data mapping principles of this approach are applicable to a range of sanification contexts, using generative processes to synthesise and perpetuate sound in real-time, mediating between designer/user/interaction and representation. Hyper-Shaku uses Evolutionary Looming to scale frequency as a consequence of input loudness and noisiness, a Neural Oscillator Network to perpetuate sounds with concordant pitch (frequency) derived from the live performer's auditory input, and gestural interaction to adjust parameters of granular synthesis and the generative processes. Auditory display is an emerging modality for data representation, both for use alone in visually heavy contexts, where sanification presents an effective alternative to visualisation, and in bi-modal audio-visual display environments where sanification can reinforce other modalities, enhancing fidelity of representation and reasoning based on it. Due to the interplay of auditory cognition, memory and the inherently time-based representation of sound, sanification can provide superior recognition to visualisation for certain types of features, such as periodicity, discrete irregularities, subtle shifts over time, stream segregation and very fine increments of data represented using frequency. This example explores a creative-context application of data (motion and breath) sanification.