Novel Distance-Based SVM Kernels for Infinite Ensemble Learning
Hsuan-Tien Lin, Ling Li · 2005
Abstract — Ensemble learning algorithms such as boosting can achieve better performance by averaging over the predictions of base hypotheses. However, most existing algorithms are limited to combining only a finite number of hypotheses, and the generated ensemble is usually sparse. It has recently been shown that the support vector machine (SVM) with a carefully crafted kernel can be used to construct a nonsparse ensemble of infinitely many hypotheses. Such infinite ensembles may surpass finite and/or sparse ensembles in learning performance and robustness. In this paper, we derive two novel kernels, the stump kernel and the perceptron kernel, for infinite ensemble learning. The stump kernel embodies an infinite number of decision stumps, and measures the similarity between examples by the ℓ1-norm distance. The perceptron kernel embodies perceptrons, and works with the ℓ2-norm distance. Experimental results show that SVM with these kernels is superior to boosting with the same base hypothesis set. In addition, SVM with these kernels has similar performance to SVM with the Gaussian kernel, but enjoys the benefit of faster parameter selection. These properties make the kernels favorable choices in practice. I.