Local adaptive algorithms for information maximization in neural networks, and application to source separation
Jeroen Dehaene, Nanayaa Twumwaa Twum-Danso · 2002
Information theoretic criteria for neural network adaptation laws have become an important focus of attention. We consider the problem of adaptively maximizing the entropy of the outputs of a deterministic feedforward neural network with real valued stochastic input signals, as considered by Bell and Sejnowski (1995). We give a new explanation for the relevance of output information (entropy) maximization for source separation applications and reinterpret Bell and Sejnowski's approach in a more general context of probability density estimation. This insight is the basis for a generalization of the approach, and we consider a family of gradient based algorithms.