Optimizing Byte-Level Representation For End-To-End ASR
Roger Wend-Huu Hsiao, Liuhui Deng, Erik McDermott, Ruchir Travadi, Xiaodan Zhuang · 2024
We propose a novel approach to optimizing a byte-level representation for end-to-end automatic speech recognition (ASR). Byte-level representation is often used by large scale multilingual ASR systems when the character set of the supported languages is large. The compactness and universality of bytelevel representation allow the ASR models to use smaller output vocabularies and therefore, provide more flexibility. UTF8 is a commonly used byte-level representation for multilingual ASR, but it is not designed to optimize machine learning tasks directly. By using auto-encoder and vector quantization, we show that we can optimize a byte-level representation for ASR and achieve better accuracy. Our proposed framework can incorporate information from different modalities, and provides an error correction mechanism. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.