Local Minima, Symmetry-breaking, and Model Pruning in Variational Free Energy Minimization
David Mackay · 2001
Approximate inference by variational free energy minimization (also known as variational Bayes, or ensemble learning) has maximum likelihood and maximum a posteriori methods as special cases, so we might hope that it can only work better than these standard methods. However, cases have been found in which degrees of freedom are ‘pruned’, perhaps inappropriately. This paper investigates this phenomenon in a toy example. Approximate inference by variational free energy minimization (also known as variational Bayes, or ensemble learning, or learning with noisy weights { see (MacKay 1995) for a review) hasmaximumlikelihoodandmaximum a posteriori methodsasspecialcases,sowemighthope that it can only work better than these standard methods. However, cases have been found in which degrees of freedom are ‘pruned’, perhaps inappropriately. This paper investigates this phenomenon in a toy example. Motivations for VFE: want to incorporate uncertainty about parameters into the modelfltting process. Also worried about the electric monastry { location in parameter space where the likelihooddiverges. Uncertainty is greatest (and singularities in the likelihood more prominent) when there is little data, so VFE is of most interest for small N. ProblemobservedbyZoubinGhahramani(studyingensemblelearningforHMMs(MacKay 1997)): extra degrees of freedom are not used. The model self-prunes. Annoying because we don’t want the pruned model, we want the model we believe in { with lots of parameters, and big error bars on them! Parameter pruning is bad news because we would like predictions to take into accoutn uncertainty. Commentonspontaneouspruning: issometimesviewedasaconvenientautomaticOccam’s razor,but does it behave correctly? Occam efiect should bevery weak for small N.