Hand-by-Hand Mentor: An AR based Training System for Piano Performance
Ruoxi Guo, Jiahao Cui, Wanru Zhao, Shuai Li, Aimin Hao · 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) · 2021
Multimedia instrument training has gained great momentum benefiting from augmented and/or virtual reality (AR/VR) technologies. We present an AR-based individual training system for piano performance that uses only MIDI data as input. Based on fingerings decided by a pre-trained Hidden Markov Model (HMM), the system employs musical prior knowledge to generate natural-looking 3D animation of hand motion automatically. The generated virtual hand demonstrations are rendered in head-mounted displays and registered with a piano roll. Two user studies conducted by us show that the system requires relatively less cognitive load and may increase learning efficiency and quality.