Information geometry and maximum likelihood criteria
WJ Byrne · 1996
This paper presents a brief comparison of two information geometries as they are used to describe the EM algorithm used in maximum likelihood estimation from incomplete data. The Alternating Minimization framework based on the I-Geometry developed by Csisz'ar is presented first, followed by the em-algorithm of Amari. Following a comparison of these algorithms, a discussion of a variation in likelihood criterion is presented. The EM algorithm is usually formulated so as to improve the marginal likelihood criterion (as described in Section 2.1). Closely related algorithms also exist which are intended to maximize different likelihood criteria. The 1-Best criterion, for example, leads to the Viterbi training algorithm used in Hidden Markov Modeling. This criterion has an information geometric description that results from a minor modification of the marginal likelihood formulation. The techniques discussed here are not given in rigorous detail, but rather at a level intended to allow comparison between them; the works cited in the bibliography should be consulted for complete and correct presentations of all methods discussed. 2 Likelihood Criteria for Incomplete Data Problems