Neural Transplant Surgery: An Approach to Pre-training Recurrent Networks
Peter W. Vamplew, Anthony I. Adams · UTAS Research Repository · 1994
Partially-recurrent networks have advantages over strictly feed-forward networks for certain spatiotemporal pattern classification or prediction tasks. However networks involving recurrent links are generally more difficult to train than their nonrecurrent counterparts. In this paper we demonstrate that the costs of training a recurrent network can be greatly reduced by initialising the network prior to training with weights 'transplanted' from a non-recurrent architecture. Introduction The approaches taken to adapting feed-forward networks to temporal processing can be divided into two main categories: non-recurrent and recurrent. Both of these approaches make use of fixed timedelays on the connections within the network to provide a means of storing temporal information. However in non-recurrent networks all such connections feed into higher layers within the network, whereas nodes in a recurrent network can also have time-delayed links to their own or lower levels. Non-recurrent ar...