Generative Replay Method Based on Integrating Random Gating and Attention Mechanism
Yao Xin He, Jing Yang, Qinglang Li, Hongchao An, Feng‐Yuan Zhang · 2023
Artificial neural networks aspire to mimic human intelligence by constantly learning from a series of tasks without forgetting past knowledge. The most typical way to achieve this kind of learning is to store previously learned task data and replay previously stored data when training a new task. However, the performance of these methods will be affected with the reduction of memory during training. To solve this problem, a new generative replay algorithm is proposed in this paper —— A generative replay method based on the fusion of random gating and attention mechanism. This method is used to generate visual significance features for the model to remember past sample information. First, we introduce an attention mechanism into the generated model to address the issue of the quality of the generated replay data. This mechanism enhances the data salience of feature information during large-scale model training and allows the generated replay network to produce visual salience features, which essentially mimic the way a human observes. Secondly, because of the overfitting phenomenon in the training process of the model, a random gating mechanism is introduced to improve the generalization of the model and improve the accuracy of the model. Finally, we use the CIFAR100 and Permuted MNIST datasets to validate the efficacy of our method.