Deriving a New Divergence Measure from Extended Cross-Entropy Error Function
Sang-Hoon Oh, Hiroshi Wakuya, Sun-Gyu Park, Hwang-Woo Noh, Jaesoo Yoo, Byung‐Won Min, Yong-Sun Oh · International Journal of Contents · 2015
Relative entropy is a divergence measure between two probability density functions of a random variable. Assuming that the random variable has only two alphabets, the relative entropy becomes a cross-entropy error function that can accelerate training convergence of multi-layer perceptron neural networks. Also, the n-th order extension of cross-entropy (nCE) error function exhibits an improved performance in viewpoints of learning convergence and generalization capability. In this paper, we derive a new divergence measure between two probability density functions from the nCE error function. And the new divergence measure is compared with the relative entropy through the use of three-dimensional plots.