Relative entropy and vector quantization
John E. Shore · The Journal of the Acoustical Society of America · 1986
Consider the problem of quantizing a vector F of measurements Fr, r = 0,…,M. In one general form of vector quantization F is quantized as the result of classification by a nearest-neighbor rule, D(F,F̂(t)) = minsεΛ D(F,F̂(s)), where D is some distortion measure and where {F(s)⋅sεΛ} is a set of predefined vectors, often called codewords. Often, the measurements Fr can be expressed as a set of expected values, ∫q† (x) fr(x)dx = Fr, and each codeword F(s) can be expressed similarly. In such cases, the relative-entropy, H(q,p) = ∫q(x)log[q(x)/p(x)]dx, can be used to define a distortion measure in (1) so that the classification is optimal in a well-defined information-theoretic sense and also computationally attractive. Furthermore, the distortion measure results in a simple method of computing codewords from training data. The method of speech coding by vector quantization is a special case of this relative-entropy method.