Prioritized Experience Replay for Continual Learning
Guannan Hu, Wu Zhang, Wenhao Zhu · 2021
Humans can learn and accumulate knowledge throughout their lifespan. Similarly, the paradigm of continual learning in artificial intelligence requires that the machine learning model preserve consolidated knowledge if a new task is adapted. However, to overcome catastrophic forgetting, a destructive issue in continual learning, memory-based approaches, replaying old experiences with experience drawn from new tasks or constraining old experiences, need a considerable memory to prevent them from decreasing consolidated knowledge, which is inefficient. To improve the efficiency of old experiences and keep memory small, we introduce prioritized experience replay, which uses a feature margin and classification margin to prioritize representative experiences. The feature margin is a cosine between original experience and average experience, and the classification margin is the correctness of the model to select experience. Experimental results show that prioritized experiences have a positive impact on alleviating catastrophic forgetting, and replaying prioritized experiences stored in a tiny reservoir, relieves overfitting and outperforms state-of-the-art continual learning approaches in a training pass.