Approximating mutual information by maximum likelihood density ratio estimation
Taiji Suzuki, Masashi Sugiyama, Jun Sese, Takafumi Kanamori · Tokyo Tech Research Repository (Tokyo Institute of Technology) · 2008
Mutual information is useful in various data processing tasks such as feature selection or independent component analysis. In this paper, we propose a new method of approximating mutual information based on maximum likelihood estimation of a density ratio function. Our method, called Maximum Likelihood Mutual Information (MLMI), has several attractive properties, e.g., density estimation is not involved, it is a single-shot procedure, the global optimal solution can be efficiently computed, and cross-validation is available for model selection. Numerical experiments show that MLMI compares favorably with existing methods.