New information theoretical approach to the storage capacity of neural networks with binary weights
Hiroki Suyari · 1999
New information theoretical approach for the storage capacities of the perceptron with binary weights w/sub i//spl isin/{0,1}, {-1, +1} are presented. Our main ideas come from the introduction of the minimum distance d between input patterns, which dominates the capacity of each neural networks. This approach by means of the new parameter d is completely different from the usual replica method in statistical physics, but it can succeed to obtain the almost same storage capacities as those by the replica method. Moreover, this information theoretical approach has some advantages of providing easier and more intuitive understanding of the capacity and the distinguishable minimum distance which characterizes the neural networks.