INTEGRATED MULTIMODAL SCORE-FOLLOWING ENVIRONMENT
Martin Ritter, Keith Hamel, Bob Pritchard · University of Michigan Library Repository · 2013
The Integrated Multimodal Score-following Environment (IMuSE) [5] is a software project aimed at developing a system for the creation, rehearsal and performance of score-based interactive computer music compositions.An enhanced version of the NoteAbilityPro [7] music notation software is the central controller in IMuSE.The score contains conventional notes and performance indications as well as discrete and continuous control messages that can be sent to other applications such as MaxMSP [15] or Pure data (Pd) [11] during performance.As well, multiple modes of score-following [2] can be used to synchronize the live performance to the score.Score-following strategies include pitch-tracking of monophonic instruments, pitchtracking and amplitude-tracking of polyphonic instruments and gesture-tracking of performers' hand movements.In this paper, we present an overview of the IMuSE system, with a focus on its abilities to monitor and coordinate multiple pitch and gesture trackers.As an example of gesture-tracking, we show how video analysis is used to follow piano hand movements.We discuss strategies for negotiating between multiple streams of score-following data and we assess the strengths, limitations and future research directions of the IMuSE system. BACKGROUNDScore-following has a relatively long history, with the first research being done in the early 1980s by Dannenberg [5,6] and Vercoe [24,25].This research was aimed at automated accompaniment where the computer would generate midi or audio tracks that were synchronized to a live musical performance.In the following decade, Puckette [17,18] developed the explode Max object that could be used for monophonic score-following of pitch-tracked or midi data.In the early 2000s, score-following systems such as suivi~ [20,21] and antescofo~ [4] were developed using Hidden Markov Models [1]; these systems prove to be far more robust in performance situations where machine perception and performer mistakes need to be accommodated.More recently, audio analysis has been used for music retrieval systems such as Query-by-Humming [12] and intelligent audio editors [7].Our focus is on score-following in live interactive computer music compositions involving several performers and where complex audio interactions between the live instruments and the computer generated sounds are required.