Enhancing Unified Streaming and Non-Streaming ASR Through Curriculum Learning With Easy-To-Hard Tasks
Yuting Yang, Yuke Li, Lifeng Zhou, Binbin Du, Haoqi Zhu · 2024
We expect a unified ASR model to deliver high performance in both streaming and non-streaming modes. However, a core challenge is that the lack of global contextual information in streaming ASR inherently hinders its performance from matching the non-streaming counterpart. Drawing inspiration from the human learning manner from easy concepts to difficult ones, we introduce a curriculum learning framework to enhance the training of unified ASR models. This framework strategically increases task complexity in a graduated, easy-to-hard order. Specifically, we develop a structured curriculum that begins with an elementary course focused on training a non-streaming model, progresses to an intermediate course for training an initial unified ASR model, and culminates in an advanced course designed to mutual promotion between these two modes via contrastive training. Experimental results on AISHELL-1 and AISHELL-2 show that our method achieves significant improvements in two modes.