A Multimodal-Based Fingering Analysis Method for Piano Playing
Wei Liao, Cheng Fan · IEEE Access · 2025
Piano performance involves complex motor skills and expressive control, making fingering analysis crucial for music education, technique correction, and intelligent accompaniment. This paper proposes a multimodal analysis framework that integrates multimodal features from expressive piano performances. Built upon a Transformer encoder, our model incorporates Position-Aware Encoding (PAE) and Dynamic Channel Attention (DCA) to better capture temporal patterns and cross-modal dependencies. Fingering identity and motion direction serve as supervision signals. A joint training objective combines cross-entropy losses with a Jensen-Shannon Divergence (JSD) term to enhance structural consistency. Experiments on an annotated dataset show our method achieves a Finger Accuracy of 0.96 and a Finger Direction Accuracy of 0.94, demonstrating strong generalization and offering new insights for intelligent education, performance feedback, and automated technique modeling.