SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check
Tuo Ji, Hang Rui Yan, Xipeng Qiu · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
Chinese Spelling Check (CSC) is to detect and correct Chinese spelling errors.Many models utilize a predefined confusion set to learn a mapping between correct characters and its visually similar or phonetically similar misuses but the mapping may be out-of-domain.To that end, we propose SpellBERT, a pretrained model with graph-based extra features and independent on confusion set.To explicitly capture the two erroneous patterns, we employ a graph neural network to introduce radical and pinyin information as visual and phonetic features.For better fusing these features with character representations, we devise masked language model alike pre-training tasks.With this feature-rich pre-training, SpellBERT with only half size of BERT can show competitive performance and make a state-of-the-art result on the OCR dataset where most of the errors are not covered by the existing confusion set