Estimating Information-Theoretic Quantities with Uncertainty Forests
Ronak Mehta, Richard Guo, Jesús Arroyo, Michael A. Powell, Hayden S. Helm, Cencheng Shen, Joshua T Vogelstein · arXiv (Cornell University) · 2019
Information-theoretic quantities, such as conditional entropy and mutual information, are critical data summaries for quantifying uncertainty. Existing estimators for these quantities either have strong theoretical guarantees or effective performance in high-dimensional data, but not both. We propose a decision forest method, Uncertainty Forests (UF), which combines quantile regression forests, honest sampling, and a finite sample correction. We prove UF provides consistent estimates for these information-theoretic quantities, including in multivariate settings. Empirically, UF reduces finite sample bias and variance in a range of both low- and high-dimensional simulated settings for estimating posterior probabilities, conditional entropies, and mutual information. In a real-world connectome application, UF quantifies the uncertainty about neuron type given various cellular features in the Drosophila larva mushroom body, a key challenge for modern neuroscience.