Memory-Free Incremental Learning on Pretrained Diffusion Model
Haohao Zhang, Jianwei Liu · 2025
Class incremental learning (CIL) aims to tackle the challenge of catastrophic forgetting when processing continuous data streams. Current state-of-the-art methods based on experience replay (ER) face limitations due to data privacy concerns and high memory usage. To mitigate this, traditional generative replay approaches train a classification network alongside a generative model, but convergence can be difficult without sufficient training data. Inspired by recent breakthroughs in large-scale pretrained generative models, we propose a Diffusion-based Generative Replay method. This approach leverages pretrained diffusion models within the generative replay framework, eliminating the need for memory buffers and avoiding the difficulties of training individual generative models from scratch. To address the slow generation speed of diffusion models, we integrate a Latent Consistency Model (LCM) module, which significantly accelerates the training process. Our framework is highly adaptable, enabling seamless integration with various incremental learning techniques. Extensive experiments on benchmark datasets demonstrate that our approach significantly outperforms traditional generative replay methods and achieves competitive performance with ER methods without relying on external memory buffers.