Minimising the Kullback-Leibler Divergence
Ralf Herbrich · 2005
where φ(x) is known as the natural statistic of x and Z(θ) := ∫ exp(θTφ(x))dx ensures normalisation. The exponential family includes many known families of distributions including the Gaussian distribution. For example, in the Gaussian case, the natural statistic φ (x) is simply the vector of all first and second moments, φ (x) = (x1, . . . , xN , x2 1 , x1x2, . . . , xN xN−1, x 2 N ). Note that the expected natural statistic of pθ(x) is given in terms of the gradient of log(Z(θ)) w.r.t. θ , that is, ∇θ log (Z (θ)) = ∫ [