StyleBERT: Chinese Pretraining by Font Style Information
Chao Lv, Han Zhang, XinKai Du, Yunhao Zhang, Ying Huang, Wenhao Li, Jia Han, Shanshan Gu · 2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC) · 2022
With the success of down streaming task using English pre-trained language model, the pre-trained Chinese language model is also necessary to get better performance of Chinese NLP tasks. Unlike the English language, Chinese has its special characters such as glyph information. So in this article, we propose the Chinese pre-trained language model named StyleBERT which incorporate the following embedding information to enhance the savvy of language model, such as word, pinyin, five stroke and chaizi. The experiments show that the model achieves well performances on a wide range of Chinese NLP tasks.