Predictive Inference, Sufficiency, Entropy and an Asymptotic Likelihood Principle
Wallace E. Larimore · Biometrika · 1983
The objective of inferring stochastic models from a set of data is to obtain the best description, by using a probability model, of the statistical behaviour of future samples of the process. A conceptual repeated sampling experiment is considered for evaluating a predictive distribution used to describe such future observations and leads to an asymptotic likelihood principle. Considerations of likelihood and sufficiency lead to the use of entropy or the Kullback–Leibler information as the natural measure of approximation to the actual distribution by a predictive distribution in repeated samples. This gives a small-sample justification for the use of entropy for evaluating parameter estimation as well as model order and structure determination procedures.