Mutual Information Estimation: Independence Detection and Consistency
Joe Suzuki · 2019
We address estimating the mutual information of variables X, Y from data. In particular, we consider a procedure that generates a sequence of contingency tables of quantized variables of X, Y, estimate the mutual information value for each of the contingency tables, and choose the largest value. This method estimates the mutual information regardless of whether each of X, Y is either discrete or continuous, and it was proved that the mutual information estimate is zero if and only if X, Y are independent, with probability one, as the sample size grows (independence detection). In this paper, we prove this method's strong consistency under a mild condition after deriving a formula of the probability that the estimate is positive when X, Y are discrete and independent. We also provide a simplified proof of independence detection using the formula.