A general formulation for learning multi-class posterior probabilities
Hongmed Ni, Tülay Adalı, Bo Wang · 2003
We use partial likelihood (PL) theory to introduce a general formulation for learning multi-class posterior probabilities. The formulation establishes a fundamental information-theoretic connection, the equivalence of partial likelihood maximization and relative entropy minimization, without making the common assumption of independent data samples. We further show that this fundamental information-theoretic relationship is satisfied for the basic class of probability models, the exponential family, which includes many important neural network probability models. Thus we provide the prospect of learning the multi-class probabilities on the PL cost using different models. We note the inefficiency of training a Softmax network and propose a modified multi-level classifier structure based on binary coding of the classes. We demonstrate the efficiency of our reduced complexity multi-level classifier by simulation results.