An entropy estimator improving mean squared error
Yasunari Yokota, M. Shiga · Electronics and Communications in Japan (Part III Fundamental Electronic Science) · 2004
Abstract Entropy estimation for a memory‐less information source has mainly been performed by the following process under the condition that the occurrence probability of each source symbol is unknown: First, the occurrence probabilities of source symbols are estimated; then the entropy is estimated by substituting the estimated occurrence probabilities into the entropy function. Such an entropy estimation is not optimum in the sense of least squares error; it causes a large error, particularly for small sample size. In this paper, we propose an entropy estimator that strictly minimizes the mean squared error for a two‐ary memory‐less information source; it is formulated as a function of the number of one type of source symbols contained in the sample set. Furthermore, we propose a technique for estimating entropy of an arbitrary M‐ary memory‐less information source by recursively applying the proposed entropy estimator for a two‐ary memory‐less information source. Numerical experiments show that this technique improves mean squared error compared to the conventional method even though the guarantee that it is the least squares error estimator has been lost. © 2004 Wiley Periodicals, Inc. Electron Comm Jpn Pt 3, 87(9): 1–10, 2004; Published online in Wiley InterScience ( www.interscience.wiley.com ). DOI 10.1002/ecjc.10163