Nonlinear backpropagation: doing backpropagation without derivatives of the activation function
John Hertz, Anders Krogh, Benny E. Lautrup, Torsten Lehmann · IEEE Transactions on Neural Networks · 1997
The conventional linear backpropagation algorithm is replaced by a nonlinear version, which avoids the necessity for calculating the derivative of the activation function. This may be exploited in hardware realizations of neural processors. In this paper we derive the nonlinear backpropagation algorithms in the framework of recurrent backpropagation and present some numerical simulations of feedforward networks on the NetTalk problem. A discussion of implementation in analog very large scale integration (VLSI) electronics concludes the paper.