Spurious Valleys in the Error Surface of Recurrent Networks—Analysis and Avoidance
Jason M. Horn, Orlando De Jesús, Martin Hagan · IEEE Transactions on Neural Networks · 2009
This paper gives a detailed analysis of the error surfaces of certain recurrent networks and explains some difficulties encountered in training recurrent networks. We show that these error surfaces contain many spurious valleys, and we analyze the mechanisms that cause the valleys to appear. We demonstrate that the principle mechanism can be understood through the analysis of the roots of random polynomials. This paper also provides suggestions for improvements in batch training procedures that can help avoid the difficulties caused by spurious valleys, thereby improving training speed and reliability.