The vanishing learning rate asymptotic for linearL2-boosting
Clément Dombry, Youssef Esstafa · ESAIM Probability and Statistics · 2024
We investigate the asymptotic behaviour of gradient boosting algorithms when the learning rate converges to zero and the number of iterations is rescaled accordingly. We mostly considerL2-boosting for regression with linear base learner as studied in P. Bühlmann and B. Yu,J. Am. Statist. Assoc.98(2003) 324–339 and analyze also a stochastic version of the model where subsampling is used at each step (J.H. Friedman,Computat. Statist. Data Anal.38(2002) 367–378). We prove a deterministic limit in the vanishing learning rate asymptotic and characterize the limit as the unique solution of a linear differential equation in an infinite dimensional function space. Besides, the training and test error of the limiting procedure are thoroughly analyzed. We finally illustrate and discuss our result on a simple numerical experiment where the linearL2-boosting operator is interpreted as a smoothed projection and time is related to its number of degrees of freedom.