A real-time responsive/interactive system for musical conducting using motion capture technology
Gary W. Hill, Jessica V. Kun · 2004
The purpose of this study was to design and implement a responsive/interactive system for conducting and conducting pedagogy. Using a Vicon 8i motion capture system and Max/MSP graphical programming environment, the system provides real-time aural and visual feedback, sensitive to the conductor's baton. The physical environment and system provide opportunities for conductors and conducting students to practice baton technique in a responsive/interactive setting that simulates a live performance “setting”. Through changes in the trajectory, analysis of velocity, size of gesture, and analysis of angles, the conductor's movements are analyzed, interpreted, and parsed to gestural segments that represent specific beats and conducting gestures executed within a bar of music. This process is repeated as the conductor moves through a musical score, while sound follows the movement, giving immediate auditory feedback in relation to beat points, tempo, dynamics and, to some extent, articulation. Simultaneous visual feedback is possible by projecting any combination of informative displays (including the notated music being conducted, dynamics, tempo, beat plane, beat number, parameters of the gestural sets, and the motion of the baton) on a screen in front of the conductor. The displays provide resources, in real-time, for metacognition and developing proprioception during the practice session. Practice sessions can be recorded and rendered for post-session analysis and evaluation of the conductor's performance from any angle in three-dimensional space, and can be captured, isolating the baton, certain parts of the body, or the entire body, depending on the focus of the conducting session. This system is the first real-time, responsive/interactive environment for conducting, using motion capture technology.