Statistical mechanics of a multilayered neural network
Eli Barkai, David Hansel, Ido Kanter · Physical Review Letters · 1990
Statistical mechanics is applied to estimate the maximal information capacity per synapse (${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$) of a multilayered feedforward neural network, functioning as a parity machine. For a large number of hidden units, K, the replica-symmetric solution overestimates dramatically the capacity, ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ensuremath{\propto}${\mathit{K}}^{2}$. However, a one-step replica-symmetry breaking gives ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ensuremath{\sim}lnK/ln2, which coincides with a theoretical upper bound. It is suggested that this asymptotic behavior is exact. Results for finite K are also discussed.