On-line Learning Dynamics in Layered Neural Networks with Arbitrary Activation Functions
Frederieke Richert, Otavio Citton, Michael L. Biehl · 2024
We revisit and extend the statistical physics based analysis of layered neural networks trained by online gradient descent.We focus on the influence of the hidden unit activation functions on the typical learning behavior in model scenarios.Expanding activation functions in terms of Hermite polynomials enables us to extend the formalism to the analysis of soft committee machines with arbitrary activation in student-teacher scenarios.This approach requires much lower computational effort than naive numerical integration, which is practically infeasible.Moreover, it now becomes possible to treat mismatched scenarios in which the student activation function differs from the one used in the target rule definition.This makes it possible to study realistic models of machine learning.