An improvement on recurrent neural network by combining convolution neural network and a simple initialization of the weights
Xu Xu, Hao Ge, Shenghong Li · 2016
Recently, the Recurrent Neural Network(RNN) has become a very powerful tool in deep learning to solve different tasks. It is considered as a viable alternative to the deep convolution neural networks. However, we all know it is a difficult task for RNN to learn long term dependencies due to vanishing and exploding gradients. The most famous and successful structure we know is Long Short Term Memory(LSTM). In this paper, we proposed a novel architecture combining Convolution Neural Network(CNN) and a variation of an RNN which is composed of Rectified Linear Units(ReLUs) and initialized with the identity matrix. We conclude that our architecture can reduce optimization time significantly and achieve a better performance compared to original neural networks on two widely-used datasets: MNIST and IMDB. Our study proves that there is still a lot of margin for the improvement of Recurrent Neural Networks.