Investigating gated recurrent neural networks for acoustic modeling
Yuanyuan Zhao, Jie Li, Shuang Xu, Bo Xu · 2016
Recurrent neural networks (RNNs) with a gating mechanism have been shown to give state-of-the-art performance in acoustic modeling, such as gated recurrent unit (GRU), long short-term memory (LSTM), long short-term memory projected (L-STMP), etc. But little is known about why these gated RNNs work and what the differences are among these networks. Based on a series of experimental comparison and analysis, we find that: a) GRU usually performs better than LSTM, for possibly GRU is able to modulate the previous memory content through the learned reset gates, helping to model the long-span dependence more efficiently for speech sequence; b) LSTMP shows comparable performance with GRU, since LSTMP has the similar ability of information selection and combination by an automatic learned linear transformation in a weight-sharing way. In experiments, a visual analysis method is adopted to understand the historical information selection mechanism in RNNs in contrast to DNN. Experimental results on three different speech recognition tasks demonstrate the above conclusions and 5%-13% relative PER or CER reduction is observed.