On Knowledge Distillation for Translating Erroneous Speech Transcriptions
Ryo Fukuda, Katsuhito Sudoh, Satoshi Nakamura · 2021
Recent studies argue that knowledge distillation is promising for speech translation (ST) using end-to-end models.In this work, we investigate the effect of knowledge distillation with a cascade ST using automatic speech recognition (ASR) and machine translation (MT) models.We distill knowledge from a teacher model based on human transcripts to a student model based on erroneous transcriptions.Our experimental results demonstrated that knowledge distillation is beneficial for a cascade ST.Further investigation that combined knowledge distillation and fine-tuning revealed that the combination consistently improved two language pairs: English-Italian and Spanish-English.