PronScribe: Highly Accurate Multimodal Phonemic Transcription From Speech and Text
Yang Yu, Matthew C Perez, Ankur Bapna, Fadi Haik, Siamak Tazari, Yu Zhang · 2023
We present PronScribe, a novel method for phonemic transcription from speech and text input based on careful finetuning and adaptation of a massive, multilingual, multimodal speech-text pretrained model.We show that our model is capable of phonemically transcribing pronunciations of full utterances with accurate word boundaries in a variety of languages covering diverse phonological phenomena, achieving phoneme error rates in the vicinity of 1-2% which is comparable to human transcribers.We show that PronScribe can effectively learn this task from relatively little training data, making it attractive even in low-resource settings.It learns from text and speech simultaneously in a coherent way, and is better than previous models using speech, text or both.Additionally, the model's good transfer learning characteristics in multilingual settings can effectively boost performance for lower-resourced languages.