Mutual Information as a Function of Moments
Wael Alghamdi, Flávio P. Calmon · 2019
We introduce a mutual information estimator based on the connection between estimation theory and information theory. By combining a polynomial approximation of the minimum mean-squared error estimator with the I-MMSE relationship, we derive a new formula for the mutual information I(X; Y) that is a function of only the marginal distribution of X, the moments of Y, and the conditional moments of Y given X. Estimating the moments in this new formula by sample moments provides an estimator of mutual information that captures desirable properties, such as being invariant under affine transformations.