Information theoretical approach to the storage capacity of neural networks with binary weights
Hiroki Suyari, Ikuo Matsuba · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1999
The storage capacity of the perceptron with binary weights w(i)in[0,1] is derived by introducing the minimum distance d between input patterns. The approach presented in this paper is based on some results in the information theory, and the obtained storage capacity 0.585 is in good agreement with the well-known value 0.59 by the replica method in statistical physics. A strength of the present information theoretical approach is that it provides an easier and more intuitive understanding for the storage capacity than the replica method, which is believed to be more reliable and informative than the Vapnik-Chervonenkis procedure.