An Empirical Analysis of Boosting Deep Networks
Sai Saketh Rambhatla, Michael Jones, Rama Chellappa · 2022 International Joint Conference on Neural Networks (IJCNN) · 2022
Boosting is a method for finding a highly accurate classifier by linearly combining many “weak” classifiers, each of which may be only moderately accurate. Thus, boosting is a method for learning an ensemble of classifiers. While boosting has been shown to be very effective for decision trees, its impact on neural networks has not been extensively studied. Using standard object recognition datasets, we verify experimentally the well-known result that a boosted ensemble of decision trees usually generalizes much better on testing data than a single decision tree with the same number of parameters. In contrast, using the same datasets and boosting algorithms, our experiments show the opposite to be true when using neural networks (both convolutional neural networks (CNNs) and multilayer perceptrons (MLPs)). We find that a single neural network usually generalizes better than a boosted ensemble of smaller neural networks with the same total number of parameters. While this is an experimental investigation, more theoretical research is warranted to understand the role of boosting in deep learning-based classifiers.