Analytical Approximations for Bayesian Inference
Tohid Ardeshiri · Linköping studies in science and technology. Dissertations · 2015
Bayesian inference is a statistical inference technique in which Bayes' theorem is used to update the probability distribution of a random variable using observations.Except for few simple cases, expression of such probability distributions using compact analytical expressions is infeasible.Approximation methods are required to express the a priori knowledge about a random variable in form of prior distributions.Further approximations are needed to compute posterior distributions of the random variables using the observations.When the computational complexity of representation of such posteriors increases over time as in mixture models, approximations are required to reduce the complexity of such representations.This thesis further extends existing approximation methods for Bayesian inference, and generalizes the existing approximation methods in three aspects namely; prior selection, posterior evaluation given the observations and maintenance of computation complexity.Particularly, the maximum entropy properties of the first-order stable spline kernel for identification of linear time-invariant stable and causal systems are shown.Analytical approximations are used to express the prior knowledge about the properties of the impulse response of a linear time-invariant stable and causal system.Variational Bayes (VB) method is used to compute an approximate posterior in two inference problems.In the first problem, an approximate posterior for the state smoothing problem for linear state-space models with unknown and timevarying noise covariances is proposed.In the second problem, the VB method is used for approximate inference in state-space models with skewed measurement noise.Moreover, a novel approximation method for Bayesian inference is proposed.The proposed Bayesian inference technique is based on Taylor series approximation of the logarithm of the likelihood function.The proposed approximation is devised for the case where the prior distribution belongs to the exponential family of distributions.Finally, two contributions are dedicated to the mixture reduction (MR) problem.The first contribution, generalize the existing MR algorithms for Gaussian mixtures to the exponential family of distributions and compares them in an extended target tracking scenario.The second contribution, proposes a new Gaussian mixture reduction algorithm which minimizes the reverse Kullback-Leibler divergence and has specific peak preserving properties.v On the 6th of November 1632, the Swedish king Gustav II Adolf, the founder of the Swedish Empire (1611-1721), was killed in the battle of Lützen in modernday Germany.Gustav II Adolf was an extremely able commander (rather than an obedient soldier), was nearsighted and had a prominent nose.It is claimed by historians that in the thick mix of gun smoke and fog covering the battlefield, he was separated from his fellow riders and killed by several shots.It is indeed just a coincidence that this thesis will be defended on the very same day 383 years later.Even the fact that I have well-known aspirations for becoming the king of Sweden does not worry me.Furthermore, I am not concerned about the fact that my opponent Dr Wolfgang Koch comes from a German defense institution which is situated only 500 km away from Lützen.Let us stay objective when forming prior beliefs.Furthermore, separation from my fellow riders is not expected to happen since I have written this thesis to enable me and my fellow riders to see through the fog of noise and smoke of disturbances using Bayes' rule.I would never have been able to write such a dissertation without the support of my fellow riders.Here, I want to acknowledge their contributions to this thesis.131 years after that fateful day reverend Thomas Bayes wrote the article "An Essay towards solving a Problem in the Doctrine of Chances " for which I am grateful.It took another 246 year till Lennart Ljung admitted me to the group and gave the opportunity to find my supervisor and my research subject.Thank you Lennart.I want to thank my supervisor Fredrik Gustafsson for his engagement in the beginning and the end of my PhD studies and his patience along the way.I want to thank the head of division of Automatic Control Svante Gunnarsson for giving me the space to maneuver and making exceptions of traditions