Analysis of Learning from Stochastic Rules in the Framework of Replica Symmetry Breaking
Tatsuya Uezu, Yoshiyuki Kabashima, Kazuo Nokura, Naomi Nakamura · Journal of the Physical Society of Japan · 1996
We study learning from examples by a perceptron when a return to an example is given by a stochastic relation which is represented by the conditional probability distribution P ( u ), where u is proportional to the inner product between the optimal synaptic weight vector and the example vector. The problem is analyzed by replica method in the case of spherical weights. Since the replica symmetric(RS) solution turns out to be unstable for stochastic cases, we consider the one-step replica symmetry breaking (RSB) solution. We investigate the asymptotic behavior of the learning curve as \(\alpha \equiv \frac{p}{N} \rightarrow \infty\), where p is the number of samples and N is the dimension of the synaptic weights. The average generalization error ε g is expressed as (ε g - ε min ) ∝α - γ in the asymptotic region. For the minimum-error algorithm, we find \(\gamma=\frac{1+\delta}{1+3\delta}\) for the RS solution and \(\gamma=\frac{1+\delta}{1+2\delta}\) for the one-step RSB solution in the case of the function P ( u ) which is monotonic and expressed as P ( u ) ≃const.\( + a \; {\rm sgn}(u)|u| ^{\delta}\) around u =0 . That is, the exponent γ is determined by the local property of P ( u ) around u =0. In particular, for the case of δ=1, the one-step RSB ansatz, unlike the RS ansatz, gives the same result up to a logarithmic correction as previous results obtained by non-replica methods in the one-dimensional model.