Knowledge Prompt for Whisper: An ASR Entity Correction Approach with Knowledge Base
Min Zhang, Xiaosong Qiao, Yanqing Zhao, Chang Su, Yinglu Li, Yuang Li, Ming Zhu, Mengyao Piao, Song Peng, Shimin Tao, Hao Yang, Yanfei Jiang · 2023
Entity correction is crucial in Automatic Speech TABLE I Recognition (ASR), since erroneous entities seriously affect our understanding of ASR results. In this paper, in order to correct entity errors, we propose a knowledge prompt approach for Whisper (a recent ASR model trained with a corpus containing 680k hours of labeled speech recorded in various conditions). For a given audio, our approach consists of three steps: (1) obtaining its ASR result by Whisper; (2) fuzzy matching the ASR result with a knowledge base to obtain candidate entities; (3) using the candidate entities as a prompt to obtain the final ASR result by Whisper again. We conduct experiments on the test dataset of open-source Chinese speech corpus AISHELLNER. Experimental results show that our approach not only significantly improves the entity recall rate in ASR results (from 70.97% to 84.82%), but also reduces the overall Character Error Rate (CER).