Sequentially Learning Multiple Meaningful Representations in Static Neural Networks
Jordi Bieger, Ida G. Sprinkhuizen-Kuyper, Iris van Rooij · 2009
Artificial neural networks (ANNs) attempt to mimic human neural networks in order to solve problems and carry out tasks. However, in contrast to their human counterparts ANNs cannot generally learn to perform new tasks without forgetting everything they already know due to a phenomenon called catastrophic interference. This paper discusses this phenomenon, shows that it occurs in multi-layer perceptrons with arbitrary task representations and proposes and discusses the static meaningful representation learning method that uses meaningful task representations to circumvent this problem when learning to perform multiple tasks. The technique is powerful enough to enable the learning of several simple tasks without changing the weights of the network. It remains to be seen whether the technique scales to more interesting task domains. The real potential of using meaningful task representations lies in their combination with other techniques.