Using spectral techniques for improved performance in artificial neural networks
Bruce E. Segee · 2002
The spectra for many common artificial neural network activation functions are derived, including members of the sigmoid family, the Gaussian function, rectangular pulses and triangular pulses. It is found that the sigmoid curves are very ill behaved in the frequency domain and thus almost always provide strong mismatch between the spectrum of the activation function and the spectrum of the function to be learned. This does not imply that networks using the sigmoid activation function cannot learn good approximations. It does imply that networks using the sigmoid activation function will learn more slowly and will be more sensitive to the loss of parameters than networks using more suitable activation functions.>