Two Approaches to Supporting Improvisational Ensemble for Music Beginners based on Body Motion Tracking
Shugo Ichinose, Souta Mizuno, Shun Shiramatsu, Tetsuro Kitahara · International Journal of Smart Computing and Artificial Intelligence · 2019
Melody recognition consists of three cognitive elements: pitch contour, rhythm, and tonality.Pitch contour and rhythm can be relatively easily represented by body motion.In comparison, tonality is difficult to understand and to represent for music beginners.In this paper, we focus on two approaches to supporting improvisational ensembles for music beginners on the basis of body motion tracking: one using a 3D motion capture camera and one using sensors in a smartphone.Users of our systems using these approaches can participate in improvisational ensembles by making hand movements that correspond to the pitch contour and rhythm without considering tonality because the generated pitch is automatically adjusted to a consonant pitch with the chords of a background tune.To deal with the delay and error in gesture recognition due to a 3D motion capture camera, we improved methods for recognizing gesture.The experimental results show that the delay of our method was improved over that of the conventional one.Furthermore, we implemented a method for motion tracking on the basis of smartphone sensors.The experimental results show the difficulties of motion tracking with smartphone sensors.Moreover, we discuss perspectives on the social reuse of improvisational melody data shared as open data.