Comparing Different Recurrent Neural Architectures on a Specific Task from Vanishing Gradient Effect Perspective
Stefano Squartini, S. Paolinelli, Francesco Piazza · 2006
The objective of this paper is to compare the performances of different Recurrent Neural systems when applied to a specific task, namely the latching problem. It is a benchmark for evaluating the impact of the Vanishing Gradient effect, arising when the Neural Network under study is trained through gradient based learning algorithms. Three distinct architectures have been addressed in different configurations: the fully Recurrent Neural Network (fRNN), the Recurrent Multiscale Network (RMN) and the Echo State Network (ESN), all already known in literature, but never considered together from this perspective. As expected, ESNs seem to be immune to the Vanishing Gradient problem, whose effect is conversely strong in the case of fRNN and partially (but significantly) mitigated when RMNs are used.