Analysis of Softmax Approximation for Deep Classifiers under Input-Dependent Label Noise
Mark Collier, Basil Mustafa, Effrosyni Kokiopoulou, Jesse Berent · arXiv (Cornell University) · 2020
Modelling uncertainty arising from input-dependent label noise is an increasingly important problem. A state-of-the-art approach for classification places a normal distribution over the softmax logits, where the mean and variance of this distribution are learned deep functions of the inputs [19]. This approach has impressive empirical performance but lacks theoretical justification. We argue that the softmax should be viewed as a smooth approximation (controlled by a temperature parameter) to an argmax in the true data generation process. Under this view, we establish a general framework for modeling input-dependent label noise with deep classifiers, whereby the state-of-the-art method [19] becomes a special case corresponding to the temperature being set to 1.0. We illustrate that the softmax temperature controls a bias-variance trade-off for the approximation and the optimal point on this trade-off is not always found at 1.0. By tuning the softmax temperature, we improve performance on image classification benchmarks with controlled label noise. For image segmentation, where input-dependent label noise naturally arises, tuning the temperature increases the mean IoU on the PASCAL VOC and Cityscapes datasets by more than 1% over the state-of-the-art model.