The mutual information of a stochastic binary channel: validity of the Replica Symmetry Ansatz
Antonio Turiel, Elka Korutcheva, Néstor Parga · 1998
We calculate the mutual information (MI) of a two-layered neural network with noiseless, continuous inputs and binary, stochastic outputs under several assumptions on the synaptic eciencies. The interesting regime corresponds to the limit where the number of both input and output units is large but their ratio is kept xed at a value . We rst present a solution for the MI using the replica technique with a replica symmetric (RS) ansatz. Then we nd an exact solution for this quantity valid in a neighborhood of = 0. An analysis of this solution shows that the system must have a phase transition at some nite value of . This transition shows a singularity in the third derivative of the MI. As the RS solution turns out to be innitely dierentiable, it could be regarded as a smooth approximation to the MI. This is checked numerically in the validity domain of the exact solution. P.A.C.S. 05.20; 87.30 Short title: Mutual Information of a Stochastic Network Journal of Physics A, 32(...