The "Softmax" Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element
Ibrahim M. Elfadel, John L. Wyatt · neural information processing systems · 1993
We use mean-field theory methods from Statistical Mechanics to derive the softmax nonlinearity from the discontinuous winner-take-all (WTA) mapping. We give two simple ways of implementing softmax as a multiterminal network element. One of these has a number of important network-theoretic properties. It is a reciprocal, passive, incrementally passive, nonlinear, resistive multiterminal element with a content function having the form of information-theoretic entropy. These properties should enable one to use this element in nonlinear RC networks with such other reciprocal elements as resistive fuses and constraint boxes to implement very high speed analog optimization algorithms using a minimum of hardware.