Bagging and Boosting
Ludmila Ilieva Kuncheva · 2004
Chapter 7 is devoted to bagging (bootstrap aggregating) and boosting, two of the most successful approaches for building classifier ensembles. Two variants of bagging are given: random forests and pasting small votes. We detail the algorithm Hedge (beta) which inspired the boosting approach. AdaBoost algorithm is presented in its multiple-class version (AdaBoost.Ml). An algorithm from the boosting group, called arc-x4, is also shown. The success of AdaBoost is explained by finding bounds on the training error for AdaBoost and by the margin theory; both are presented in this chapter. Bias-variance decomposition of the classification error is perceived as a useful analysis tool for explaining the success of bagging and boosting. Three bias-variance decomposition approaches are given.