On the bootstrap approach for support vector machines and related kernel based methods
Andreas Christmann, Robert Hable · 2013
Kernel based methods and in particular support vector Machines (SVMs) based on a general loss function and on a general kernel play an important role in statistical machine learning for many rea-sons. Such general SVMs can be considered as certain Hilbert-space valued kernel based regularized M-estimators. If some weak assumptions are satisfied, such SVMs are solutions of a well-posed mathe-matical problem in Hadamard’s sense (i.e., there exists a unique solution which continuously depends on the data), are universally consistent with good learning rates, are statistically stable with respect to many notions of statistical robustness, and are attractive from a computational point of view. Last but not least, such kernel based methods have demonstrated their good generalization properties in many large scale applications with an unknown high-dimensional dependency structure. The question how to compute a good approximation of the finite sample distribution of such kernel based methods is not yet fully addressed in the literature. From an applied point of view, this might be considered as a serious gap, because knowledge of the finite sample distribution or the appropri-ateness of an asymptotic distribution is the basis to draw statistical decisions like confidence regions, prediction intervals, tolerance intervals or tests. Here we use the empirical bootstrap (Efron, 1979) and show that this approach provides a consistent estimator for the distribution of SVMs under some relatively mild conditions.