Adversarial Training for Uncertainty Estimation in Cross-Lingual Text Classification
Xia Lina, Askar Hamdulla, Mijit Ablimit, Sijie Wang · 2024
Multilingual pre-trained models have achieved remarkable performance in cross-lingual transfer tasks, but their effectiveness heavily depends on the amount of labeled data available for training. Recent research have demonstrated that self-training for semi-supervised learning can effectively improve deep learning models by utilizing unlabeled data in the presence of limited training data. In this paper, we propose a neural network model with heteroscedastic uncertainty estimation based on adversarial training. The task model for cross-lingual learning consists of a multilingual pre-trained encoder and a dual-channel feature extraction layer, enhancing the model’s ability to model contextual features. In the self-training framework with adversarial perturbations, we utilize pseudo-labeled data for dynamic iterative training. By combining uncertainty estimation and adversarial training, we selectively choose representative samples from unlabeled data, mitigating the issue of label noise propagation and benefiting model training. The proposed approach is evaluated on two cross-lingual datasets, MLDoc and PAWS-X, and experimental results demonstrate the effectiveness of our method.