Recurrent Neural Networks (RNN)
Arash Gharehbaghi · 2023
Recurrent neural networks gain increasing popularity in modelling and analysis of time-dependent information of dynamic processes. This chapter focuses on (local) recurrent networks which contain dynamic inner feedback despite the globally feedforward connections between neurons. First, the basic principles of structure unfolding in time and parameter sharing in network are presented, followed by the elaboration of the Backpropagation through Time (BATT) algorithm for learning of the networks. The gradient vanishing problem is explained for the situations of learning of long-term dependency. Then the long-short term memory (LSTM) cell and gated recurrent unit (GRU) are introduced respectively, which can be used to replace usual hidden units in recurrent networks to overcome the difficulty of capturing long-term dependency. Moreover, a short discussion is given to echo state networks (ESNs), which only allow for learning of the output layer while making the input and recurrent weights nontrainable.