Optimization of Barron density estimates
Igor Vajda, Edward C. van der Meulen · IEEE Transactions on Information Theory · 2001
We investigate a nonparametric estimator of the probability density introduced by Barron (1988, 1989). Earlier papers established its consistency in a strong sense, e.g., in the expected information divergence or expected chi-square divergence. This paper pays main attention to the expected chi-square divergence criterion. We give a new motivation of the Barron estimator by showing that a maximum-likelihood estimator (MLE) of a density from a family important in practice is consistent in expected information divergence but not in expected chi-square divergence. We also present new and practically applicable conditions of consistency in the expected chi-square divergence. Main attention is paid to optimization (in the sense of the mentioned criterion) of the two objects specifying the Barron estimator: the dominating probability density and the decomposition of the observation space into finitely many bins. Both problems are explicitly solved under certain regularity assumptions about the estimated density. A simulation study illustrates the results in exponential, Rayleigh, and Weibull families.