Instance-adaptive training with noise-robust losses against noisy labels
Lifeng Jin, Linfeng Song, Kun Xu, Dong Yu · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
In order to alleviate the huge demand for annotated datasets for different tasks, many recent natural language processing datasets have adopted automated pipelines for fast-tracking usable data.However, model training with such datasets poses a challenge because popular optimization objectives are not robust to label noise induced in the annotation generation process.Several noise-robust losses have been proposed and evaluated on tasks in computer vision, but they generally use a single dataset-wise hyperparamter to control the strength of noise resistance.This work proposes novel instance-adaptive training frameworks to change dataset-wise hyperparameters of noise resistance in such losses to be instance-specific.Such instance-specific noise resistance hyperparameters are predicted by special instance-level label quality predictors, which are trained along with the main models.Experiments on noisy and corrupted NLP datasets show that proposed instance-adaptive training frameworks help increase the noiserobustness provided by such losses, promoting the use of the frameworks and associated losses in training NLP models with noisy data.