Unsupervised and Supervised Clustering: The Mutual Information between Parameters and Observations
Didier Herschkowitz, Jean‐Pierre Nadal · Neural Information Processing Systems · 1998
Recent works in parameter estimation and neural coding have demonstrated that optimal performance are related to the mutual information between parameters and data. We consider the mutual information in the case where the dependency in the parameter (a vector θ) of the conditional p.d.f. of each observation (a vector ξ, is through the scalar product θξ only. We derive bounds and asymptotic behaviour for the mutual information and compare with results obtained on the same model with the replica technique.