Joint Encoder-Decoder Self-Supervised Pre-training for ASR
A Arunkumar, Srinivasan Umesh · Interspeech 2022 · 2022
Self-supervised learning (SSL) has shown tremendous success in various speech-related downstream tasks, including Automatic Speech Recognition (ASR).The output embeddings of the SSL model are treated as powerful short-time representations of the speech signal.However, in the ASR task, the main objective is to get the correct sequence of acoustic units, characters, or byte-pair encodings (BPEs).Usually, encoderdecoder architecture works exceptionally well for a sequenceto-sequence task like ASR.Therefore, in this paper, we propose a new paradigm that exploits the power of a decoder during self-supervised learning.We use Hidden Unit BERT (Hu-BERT) SSL framework to compute the conventional masked prediction loss for the encoder.In addition, we have introduced a decoder in the SSL framework and proposed a target preparation strategy for the decoder.Finally, we use a multitask SSL setup wherein we jointly optimize both the encoder and decoder losses.We hypothesize that the presence of a decoder in the SSL model helps it learn an acoustic unit-based language model, which might improve the performance of an ASR downstream task.We compare our proposed SSL model with HuBERT and show up to 25% relative improvement in performance on ASR by finetuning on various LibriSpeech subsets.