Ensemble learning for prediction
Jerome H. Friedman, Bogdan E. Popescu · 2004
The goal of this dissertation is to study and develop automatic prediction technology that is accurate, fast and interpretable. The focus here is on decision tree ensembles methods and extensions. Characteristics of popular ensemble methods such as bagging, random forests and boosting are examined and leveraged to create new predictive methodology. The classic ensembles are integrated in an unifying paradigm, the Important Sampled Learning Ensembles. This framework explains some of the properties of these ensembles and suggests modifications that can significantly enhance their accuracy while dramatically improving their computational performance. The ISLES are two-stage algorithms having at the front-end a base learners ensemble generation routine followed by post-processing algorithms that perform a fast gradient directed regularized fit for regression, robust regression and classification. The post-processing algorithms developed here can also serve as a stand-alone toolkit for fitting large linear systems. Decision tree ensembles can generate rules that are fit together with the gradient directed regularized linear algorithms, leading to accurate and interpretable RuleFit models. ISLE and RuleFit are flexible methodologies, able to automatically handle non-linearities and interactions, mixtures of categorical and continuous variables with missing data, as well as feature selection.