Estimating the entropy of discrete distributions
András Antos, Ioannis Kontoyiannis · 2002
Given an i.i.d. sample (X/sub 1/,...,X/sub n/) drawn from an unknown discrete distribution P on a countably infinite set, we consider the problem of estimating the entropy of P. We show that the plug-in estimate is universally consistent and that, without further assumptions, no rate of convergence results can be obtained for any sequence of entropy estimates. Under additional conditions we get convergence rates for the plug-in estimate and for an estimate based on match-lengths. The behavior of the expected error of the plug-in estimate is shown to be in sharp contrast to the finite-alphabet case.