Mixture models achieving optimal coding regret
Andrew R. Barron, Jun’ichi Takeuchi · 2002
Summary form only given. The relative entropy between the Jeffreys mixture and Shtarkov's normalized maximum likelihood distribution tends to zero as the sample size gets large for smooth parametric families of distributions, and equivalently, the Jeffreys mixture is asymptotically maximin for the coding regret problem. Implications of this fact reveal lower bounds, including explicit constants, that hold no matter what coding strategy is used for most sequences generated by most distributions in the family. However, for a small set of sequences the Jeffreys mixture and the normalized maximum likelihood do not agree asymptotically. That is, the Jeffreys mixture is not asymptotically minimax. This discrepancy occurs not only for sequences with maximum likelihood parameter values near the boundary, but also, when the parametric family is not of exponential type, for sequences for which the empirical Fisher information is not close to the Fisher information at the MLE. We show how the non-exponential parametric families can be modified by an exponential tilting using the empirical Fisher information to define a somewhat higher-dimensional family. A mixture defined on the extended family, with most of the mass on the original family, is shown to have asymptotically minimax coding regret.