Estimating Renyi Entropy of Discrete Distributions
Jayadev Acharya, Alon Orlitsky, Ananda Theertha Suresh, Himanshu Tyagi · IEEE Transactions on Information Theory · 2016
It was shown recently that estimating the Shannon entropy H(p) of a discrete k-symbol distribution p requires Θ(k/log k) samples, a number that grows near-linearly in the support size. In many applications, H(p) can be replaced by the more general Rényi entropy of order α and Hα(p). We determine the number of samples needed to estimate Hα(p) for all α, showing that α1/αsamples, noninteger α > 1 requires a near-linear k samples, but, perhaps surprisingly, integer α > 1 requires only Θ(k1-1/α) samples. Furthermore, developing on a recently established connection between polynomial approximation and estimation of additive functions of the form Σxf (px), we reduce the sample complexity for noninteger values of α by a factor of log k compared with the empirical estimator. The estimators achieving these bounds are simple and run in time linear in the number of samples. Our lower bounds provide explicit constructions of distributions with different Rényi entropies that are hard to distinguish.