The weight-space of the binary perceptron
Robert W. Penney, David C. Sherrington · Journal of Physics A Mathematical and General · 1993
With a view to finding features of the weight-space of the binary perceptron that might be instructive for training binary-synapse neural networks, the maximally-stable perceptron having binary-valued weights is compared with continuous-weight perceptrons, for universal choices of stored patterns. The fraction of synaptic-weights correctly predicted by clipping the synapses of the continuous network is calculated in the thermodynamic limit and compared with simulation results for smaller systems. Numerical experiments show good agreement with theory but, in addition, indicate that those binary synapses likely to be wrongly predicted by weight-clipping are predominantly those which are weakest in the continuous-synapse perceptron. Although not rescuing training time from growing exponentially in the system size, our results suggest ways of significantly accelerating the search for successful, albeit possibly imperfect, neural networks with discrete-valued couplings.