RoChBert: Towards Robust BERT Fine-tuning for Chinese
Zihan Zhang, Jinfeng Li, Ning Shi, Bo Yuan, Xiangyu Liu, Rong Zhang, Hui Xue, Donghong Sun, Chao Zhang · 2022
Despite of the superb performance on a wide range of tasks, pre-trained language models (e.g., BERT) have been proved vulnerable to adversarial texts.In this paper, we present RoChBert, a framework to build more Robust BERT-based models by utilizing a more comprehensive adversarial graph to fuse Chinese phonetic and glyph features into pretrained representations during fine-tuning.Inspired by curriculum learning, we further propose to augment the training dataset with adversarial texts in combination with intermediate samples.Extensive experiments demonstrate that RoChBert outperforms previous methods in significant ways: (i) robust -RoChBert greatly improves the model robustness without sacrificing accuracy on benign texts.Specifically, the defense lowers the success rates of unlimited and limited attacks by 59.43% and 39.33% respectively, while remaining accuracy of 93.30%; (ii) flexible -RoChBert can easily extend to various language models to solve different downstream tasks with excellent performance; and (iii) efficient -RoChBert can be directly applied to the fine-tuning stage without pre-training language model from scratch, and the proposed data augmentation method is also low-cost. 1