Neural Computation with Winner-Take-All as the Only Nonlinear Operation
Wolfgang Maass · 1999
Everybody "knows" that neural networks need more than a single layer of nonlinear units to compute interesting functions. We show that this is false if one employs winner-take-all as nonlinear unit: ffl Any boolean function can be computed by a single k-winner-take-all unit applied to weighted sums of the input variables. ffl Any continuous function can be approximated arbitrarily well by a single soft winner-take-all unit applied to weighted sums of the input variables. ffl Only positive weights are needed in these (linear) weighted sums. This may be of interest from the point of view of neurophysiology, since only 15% of the synapses in the cortex are inhibitory. In addition it is widely believed that there are special microcircuits in the cortex that compute winner-take-all. ffl Our results support the view that winner-take-all is a very useful basic computational unit in Neural VLSI: 2 it is wellknown that winner-take-all of n input variables can be computed very efficiently...