Towards Stable Natural Language Understanding via Information Entropy Guided Debiasing
Li Du, Xiao Hua Ding, Zhouhao Sun, Ting Liu, Bing Qin, Jingshuo Liu · 2023
Although achieving promising performance, current Natural Language Understanding models tend to utilize dataset biases instead of learning the intended task, which always leads to performance degradation on out-of-distribution (OOD) samples.To increase the performance stability, previous debiasing methods empirically capture bias features from data to prevent the model from corresponding biases.However, our analyses show that the empirical debiasing methods may fail to capture part of the dataset biases and mistake semantic information of input text as biases, which limits the effectiveness of debiasing.To address these issues, we propose a debiasing framework IEGDB that comprehensively detects the dataset biases to induce a set of biased features, and purify the biased features with the guidance of information entropy.Experimental results show that IEGDB can consistently improve the stability of performance on OOD datasets for a set of widely adopted NLU models.