Maximizing information about a noisy signal with a single non-linear neuron
James Orwell · 1999
For noise-free information maximization, the output signal entropy must be maximized. This is not true for a noisy input: rather, it must be the difference between this entropy and the residual output uncertainty. A definition of information density is introduced, which provides a discrete local measure of bandwidth efficiency. Novel training rules are proposed which enforce a uniformity of this density. This predicts a different optimal transfer function, from that which follows from the maximization of output entropy alone. It is shown to provide higher information transmission properties on real and synthetic data.