KM-Prompt: A Prompt-Based CNN for Continual Learning
Junwei Chen, Zhigang Zeng · 2024
Deep neural networks have achieved remarkable success in many applications but often suffer from catastrophic forgetting—a significant drop in performance on previously learned tasks when training on new ones. Continual learning seeks to address this challenge. A straightforward yet effective solution is to replay a subset of past data, though this strategy increases memory costs and can raise concerns over data privacy. Recently, prompt-based methods have gained considerable attention, as they can achieve strong performance without requiring a rehearsal buffer. In this work, we are the first to introduce a prompt-based method to the CNN framework, breaking the previous reliance on Transformers. By leveraging kernel prompts and mask prompts to tune the CNN backbone, our model captures task-specific patterns while filtering out irrelevant information from feature maps. Prompts are combined based on the similarity between features and queries, enabling dynamic selection of knowledge. Extensive experiments demonstrate the superiority of our method, with improvements of over 0.8% and 7% in average final accuracy on CIFAR-100 and ImageNet-R, respectively, compared to classic methods.