Average Top-k Aggregate Loss for Supervised Learning
Siwei Lyu, Yanbo Fan, Yiming Ying, Bao-Gang Hu · IEEE Transactions on Pattern Analysis and Machine Intelligence · 2020
In this work, we introduce theaverage top-$k$k($\mathrm {AT}_k$) loss, which is the average over the$k$largest individual losses over a training data, as a new aggregate loss for supervised learning. We show that the$\mathrm {AT}_k$loss is a natural generalization of the two widely used aggregate losses, namely the average loss and the maximum loss. Yet, the$\mathrm {AT}_k$loss can better adapt to different data distributions because of the extra flexibility provided by the different choices of$k$. Furthermore, it remains a convex function over all individual losses and can be combined with different types of individual loss without significant increase in computation. We then provide interpretations of the$\mathrm {AT}_k$loss from the perspective of the modification of individual loss and robustness to training data distributions. We further study the classification calibration of the$\mathrm {AT}_k$loss and the error bounds of$\mathrm {AT}_k$-SVM model. We demonstrate the applicability of minimum average top-$k$learning for supervised learning problems including binary/multi-class classification and regression, using experiments on both synthetic and real datasets.