Text-to-image Diffusion Model Suppressing Catastrophic Forgetting via Elastic Weight Consolidation
Haruka Matsuda, Ren Togo, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama · 2023
This paper presents a personalized text-to-image diffusion model for multiple contents introducing Elastic Weight Consolidation (EWC) suppressing catastrophic forgetting. The proposed method can support personalization to multiple contents based on continuous learning, while conventional methods mainly focus on fine-tuning of a single object. Since the standard continuous learning has a problem of catastrophic forgetting due to the training without considering the importance of the previous tasks, we address this issue by adding a regularization term of EWC in the second and later fine-tuning. By introducing the EWC, the proposed method enables the suppression of the catastrophic forgetting in the text-to-image generation tasks. Experimental results show that the proposed method is effective for the catastrophic forgetting based on Frechet Inception Distance (FID) as a quantitative measurement.