Separator-Transducer-Segmenter: Streaming Recognition and Segmentation of Multi-party Speech
Ilya Sklyar, Anna Piunova, Christian Osendorfer · Interspeech 2022 · 2022
Streaming recognition and segmentation of multi-party conversations with overlapping speech is crucial for the next generation of voice assistant applications.In this work we address its challenges discovered in the previous work on multi-turn recurrent neural network transducer (MT-RNN-T) with a novel approach, separator-transducer-segmenter (STS), that enables tighter integration of speech separation, recognition and segmentation in a single model.First, we propose a new segmentation modeling strategy through start-of-turn and end-of-turn tokens that improves segmentation without recognition accuracy degradation.Second, we further improve both speech recognition and segmentation accuracy through an emission regularization method, FastEmit, and multi-task training with speech activity information as an additional training signal.Third, we experiment with end-of-turn emission latency penalty to improve end-point detection for each speaker turn.Finally, we establish a novel framework for segmentation analysis of multi-party conversations through emission latency metrics.With our best model, we report 4.6% abs.turn counting accuracy improvement and 17% rel.word error rate (WER) improvement on Lib-riCSS dataset compared to the previously published work.