Dynamics of gradient-based learning and applications to hyperparameter estimation
K. Y. Michael Wong, Peixun Luo, Fuli Li · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · 2003
We analyse the dynamics of gradient-based learning algorithms using the cavity method, considering the cases of batch learning with non-vanishing rates, and on-line learning. It has an an excellent agreement with simulations. Applications to efficient and precise estimation of hyperparameters are proposed.