Multi-Task Output Space Regularization
Sergey Feldman, Béla A. Frigyik, Maya R. Gupta, Luca Cazzanti, Peter Sadowski · 2011
We investigate multi-task learning from an output space regularization perspective. Most multi-task approaches tie together related tasks by constraining them to share input spaces and function classes. In contrast to this, we propose a multi-task paradigm which we call output space regularization, in which the only constraint is that the output spaces of the multiple tasks are related. We focus on a specific instance of output space regulariza-tion, multi-task averaging, that is both widely applicable and amenable to analysis. The multi-task averaging estimator improves on the single-task sample average under certain conditions, which we detail. Our analysis shows that for a simple case the optimal simi-larity depends on the ratio of the task variance to the task differences, but that for more complicated cases the optimal similarity behaves nonlinearly. Further, we show that the estimates produced are a convex combination of the tasks ’ sample averages. We discuss the Bayesian viewpoint. Three applications of multi-task output space regularization are presented: multi-task kernel density estimation, multi-task-regularized empirical moment constraints in similarity discriminant analysis, and multi-task local linear regression. Ex-periments on real data sets show statistically significant gains.