Piano Performance Analysis Using Technique Information Extracted from Videos
Samuel Cantor · Zenodo (CERN European Organization for Nuclear Research) · 2023
Piano technique is an important aspect of the performance and learning of the mu-sical instrument and requires an understanding of the physical and visual processes of performance. Many of the elements of a performance will be influenced by the pianist’s technique, which can include limitations such as their skill, hand size, and finger dexterity. As such, understanding technique can enhance many applications in piano, including skill assessment and movement suggestions for education, more realistic generative models based on the constraints/movement of hands, refining the classification range in polyphonic pitch estimation by modelling the hand placement on the piano and estimating the most efficient or desirable fingers for a sequence of notes, among many other applications. We present a generalized pipeline for extracting hand movement information from piano performance videos and demon-strate the creation of a dataset from YouTube videos. An analysis is performed on the created dataset and outlines the accuracy in extracting technique information using the given hand-pose estimation and transcription models. We then train an autoregressive graph neural network using this new dataset for the task of Auto-matic Piano Fingers (APF) and benchmark the performance in comparison with state-of-the-art models. Limitations to automatic technique extraction from videos are discussed, and a series of feature improvements are documented to improve the next iterations of this dataset.