Beyond the Noise: The Simple Power of Negative Label Smoothing
Yun Zhou, Hao Yang · 2023
Label Smoothing (LS) is widely used during model training to make their predictions more reliable and less overconfident. However, when applying these models to new, unlabelled and noisy data (a process called test time domain adaptation, or TTA), LS isn’t as effective. This is because noisy data has high entropy, meaning LS might make the model’s predictions too ’smooth’ or uncertain. In this paper, we introduce Negative Label Smoothing (NLS). This method works differently from LS by negatively adjusting the balance between certain (hard) and uncertain (soft) labels, which strengths model decision and is helpful for model optimization. We found that NLS is more effective in TTA scenarios and experiments show that this new strategy improves existed model performance during TTA.