Learning algorithm that gives the Bayes generalization limit for perceptrons
Osame Kinouchi, Nestor Caticha · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1996
A variational approach to the study of learning a linearly separable rule by a single-layer perceptron leads to a gradient descent learning algorithm with exactly the same generalization ability as the Bayes limit calculated by Opper and Haussler [Phys. Rev. Lett. 66, 2677 (1991)]. This is done by finding, through the Gardner-Derrida replica method, the student-teacher overlap $R$ as a functional of the algorithm cost function and maximizing this functional. The resulting cost function is closely related to the optimal cost function derived for on-line learning.