Statistical Mechanics of On-Line Learning Using Correlated Examples and Its Optimal Scheduling
Takashi Fujii, Hidetaka Ito, Seiji Miyoshi · Journal of the Physical Society of Japan · 2017
We theoretically study the generalization capability of on-line learning using several correlated input vectors in each update in a statistical-mechanical manner. We consider a model organized with linear perceptrons with Gaussian noise. First, in a noiseless case, we analytically derive the optimal learning rate as a function of the number of examples used in one update and their correlation. Next, we analytically show that the use of correlated examples is effective if the optimal learning rate is used, even when there is some noise. Furthermore, we propose a novel algorithm that raises the generalization capability by increasing the number of examples used in one update with time.