CATASTROPHIC FORGETTING IN NEURAL NETWORKS
John R Riesenberg · OhioLink ETD Center (Ohio Library and Information Network) · 2000
This paper explores the phenomenon of 'catastrophic forgetting' in artificial neural networks (ANNs).Catastrophic forgetting simply is the inability of an ANN to learn a second set of information without forgetting what it previously learned.For the past decade, catastrophic forgetting or, as it is sometimes called 'catastrophic interference', has baffled many researchers trying to solve this problem.McCloskey and Cohen (1989) attempted to recreate the Barnes and Underwood (1959) study of retroactive inhibition with an ANN simulation, but quickly found that their backpropagation ANN model did not simulate human learning very well.They found that because knowledge is highly distributed throughout the network, learning a second set of items causes the network to create a new solution space based on the newly learned information (McCloskey and Cohen, 1989).This approach mistakenly led many to believe that solving the catastrophic forgetting problem involved eliminating overlapping hidden representations in the network.Robins (1995) developed the sweep pseudorehearsal procedure by viewing a trained network as a kind of function approximator.This approach unfortunately lacked plausibility as a cognitive model.French (1997) and Ans and Rousset (1997) each built dual-network architectures based on the evidence gathered by McClelland, McNaughton, and O'Reilly (1995) that the brain solved the catastrophic forgetting problem by evolving the hippocampus and neocortex into complementary learning systems.These dual-network architectures used the sweep pseudorehearsal technique to pass a network's knowledge to another network to overcome the catastrophic forgetting in backpropagation networks.To prepare the reader for the technical discussions about catastrophic forgetting, three popular ANN models are presented (backpropagation, Hopfield, and SDM) to give a flavor of how ANN models work.ACKNOWLEDGMENTS I would like to thank: Dr. Honeck for his guidance and support in helping me complete this program, Dr. Chiu for his contributions on this subject, Lynn Yosua, my wife, who with love