Notes on PCA, Regularization, Sparsity and Support Vector Machines
Tomaso Poggio, Federico Girosi · DSpace@MIT (Massachusetts Institute of Technology) · 1998
We derive a new representation for a function as a linear combination of local correlation kernels at optimal sparse locations and discuss its relation to PCA, regularization, sparsity principles and Support Vector Machines. We also discuss its Bayesian interpretation and justification. We first review previous results for the approximation of a function from discrete data (Girosi, 1998) in the context of Vapnik's feature space and dual representation (Vapnik, 1995). We apply them to show 1) that a standard regularization functional with a stabilizer defined in terms of the correlation function induces a regression function in the span of the feature space of classical Principal Components and 2) that there exist a dual representations of the regression function in terms of a regularization network with a kernel equal to a generalized correlation function. We then describe the main observation of the paper: the dual representation in terms of the correlation function can be sparsified ...