Unsupervised and supervised learning: Mutual information between parameters and observations
Didier Herschkowitz, Jean‐Pierre Nadal · Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics · 1999
We study the mutual information between parameter and data for a family of supervised and unsupervised learning tasks. The parameter is a possibly, but not necessarily, high-dimensional vector. We derive exact bounds and asymptotic behaviors for the mutual information as a function of the data size and of some properties of the probability of the data given the parameter. We compare these exact results with the predictions of replica calculations. We briefly discuss the universal properties of the mutual information as a function of data size.