Expectation Propagation for Exponential Families
Matthias Seeger · 2005
This is a tutorial describing the Expectation Propagation (EP) algorithm for a general exponential family. Our focus is on simplicity of exposition. Although the overhead of translating a specific model into its exponential family representation can be considerable, many apparent complications of EP can simply be sidestepped by working in this canonical representation. Note: This material is extracted from the Appendix of my PhD thesis (see www.kyb.tuebingen.mpg.de/bs/people/seeger/papers/thesis.html). 1 Exponential Families Definition 1 (Exponential Family) A set F of distributions with densities P (x|θ) = exp ( θ T φ(x) − Φ(θ) ) , θ ∈ Θ, Φ(θ) = log exp ( θ T φ(x) ) dµ(x) w.r.t. a base measure µ is called an exponential family. Here, θ are called natural parameters, Θ the natural parameter space, φ(x) the sufficient statistics, and Φ(θ) is the log partition function. Furthermore, η = Eθ[φ(x)] are called moment parameters, where Eθ[·]