Dynamic neural networks, comparing spiking circuits and LSTM
Arne Koopman, Matthijs van Leeuwen, Jilles Vreeken · Utrecht University Repository (Utrecht University) · 2003
this memory gave rise to fundamental problems during the training phase of siginoid recurrent networks. Popular training algorithms for recurrent neural networks include Back-Propagation Through Time (BPTT) and Real-Time Recurrent Learning (RTRL) [9,10,12]. During the learning phase, BPTT gradually enfolds each layer of the network into a multi-layer network, in which each layer represents a snapshot of the corresponding time step. The resulting network allows the error to flow in time and is used for learning temporal correlations. The temporal error is provided in a way similar to that of the well known back-propagation algorithm [29]. A major drawback of BPTT is its need to record the whole network state, inputs, target vectors and weights during the training phase, as weight adjustment is done only after the epoch has ended. In contrast, RTRL allows for real-time weight adjustments, at the cost of losing the ability to follow the true gradient, which gives no practical limitations though [9]