Advanced Long-Content Speech Recognition With Factorized Neural Transducer
Xun Gong, Yu Wu, Jinyu Li, Shujie Liu, Rui Zhao, Xie Chen, Yanmin Qian · IEEE/ACM Transactions on Audio Speech and Language Processing · 2024
Long-form automatic speech recognition (ASR) has obtained increasing interest in recent years, as it captures the relationship among consecutive historical sentences while decoding the current sentence. In this paper, we propose two novel approaches, which integrate long-form information into the factorized neural transducer (FNT) based architecture in both non-streaming (referred to asLongFNT) and streaming (referred to asSLongFNT) scenarios. We first investigate whether long-form transcriptions can improve the vanilla conformer transducer (C-T) models. Our experiments indicate that the vanilla C-T models do not exhibit improved performance when utilizing long-form transcriptions, possibly due to the predictor network of C-T models not functioning as a pure language model. Instead, FNT shows its potential in utilizing long-form information, where we propose theLongFNTmodel and explore the impact of long-form information in both text (LongFNT-Text) and speech (LongFNT-Speech). The proposed LongFNT-Text and LongFNT-Speech models further complement each other to achieve better performance, with transcription history proving more valuable to the model. The effectiveness of our LongFNT approach is evaluated on LibriSpeech and GigaSpeech corpora, and obtains relative 19% and 12% word error rate reduction, respectively. Furthermore, we extend the LongFNT model to the streaming scenario, which is namedSLongFNT, consisting of SLongFNT-Text and SLongFNT-Speech approaches to utilize long-form text and speech information. Experiments show that the proposed SLongFNT model achieves relative 26% and 17% WER reduction on LibriSpeech and GigaSpeech respectively while keeping a good latency, compared to the FNT baseline. Overall, our proposedLongFNTandSLongFNThighlight the significance of considering long-form speech and transcription knowledge for improving both non-streaming and streaming speech recognition systems.