Where is the beat : introducing 'AV-PhonemeBeatSync', a multimodal singing dataset aiming at understanding coarticulation and rhythm in singing
Thomas Le Roux · Zenodo (CERN European Organization for Nuclear Research) · 2023
Singing, the vocal production of musical tones is a key element in music. It plays a central role in the emotions perceived while listening to a musical piece, and has been present in human culture since antiquity, acting as a way to transmit information such as stories, rituals or tales from the past. Being able to sing well is considered as a valuable skill to have, and singing has shown benefits in mental and physical health. A distinctive area of Music Information Retrieval is dedicated to the analysis and synthesis of singing. Research in this domain focuses notably on understanding the singing processus, assessing performance quality, or generating automatic singing content. While most of the approaches now uses extensive machine learning techniques, datasets are lacking, and there is none containing annotated audiovisual singing performances and none containing clear beat annotations. Previous research has been done on analyzing a singing performance and giving feedback, but none of them tried to understand where the performers line the syllables with their percep-tion of the beat, and where the emphasis is placed. In this Master Thesis, we present a new multimodal dataset : AV_PhonemeBeatSync, aiming at providing new annotated data to propel various research on singing anal-ysis and synthesis. A brief analysis is conducted on the gathered data, and we find that performers, when having to sing a syllable, concentrate their attention on the vowels and disregard the consonants, regardless of their singing experience. AV_PhonemeBeatSync contains valuable data that is worth mining, and could be used to train and evaluate neural networks aiming at visually understanding the coarticulation phenomenon, and the relationship between lyrics and rhythm.