Subnetwork Knowledge Injection and Transferable Parameter Updating Strategy for Continual Learning of Vision-and-Language Tasks
Jiong Wang, Qing Zhang, Jie Liu, Kaifeng Nie, Ze Zhang, Linqi Song, Yinqiao Li · IEEE Transactions on Artificial Intelligence · 2025
The external environment of the real world is often filled with various types of multimodal information that are highly dynamic and unpredictable. Therefore, intelligent systems need to continuously learn and retain knowledge. Continual learning (CL) provides a foundation for vision-and-language (VaL) models to adapt to the changes in the real world. Existing research on CL for VaL models mainly focus on preserving memory stability to overcome catastrophic forgetting. However, it is difficult for existing methods to flexibly adapt to the dynamic growth of external information. In this paper, we propose a generic approach that appropriately attenuates and employs old memories in parameter distributions to improve learning plasticity. We propose a new parameter-sharing CL method that combines adaptive parameter update strategies with network pruning to enhance the plasticity of VaL models during the continual learning process. When learning a new task, we utilize the pruned model for learning new tasks. And then we employ the sub-network parameter initialization updating strategy to transfer the most similar important knowledge from learned tasks. Subsequently, to strength the stability of the model, we further update the model parameters, adjusting old memories to better adapt to new task learning. Experiments on a series of VaL tasks have shown that our proposed method outperforms the compared methods in improving model plasticity, and is more stable than existing parameter sharing methods.