ConWave-LoRA: Concept Fusion in Customized Diffusion Models with Contrastive Learning and Wavelet Filtering
Xinying Liu, Xiaogang Huo, Zhihui Yang · Computers · 2025
Customizing diffusion models via Low-Rank Adaptation (LoRA) has become a standard approach for customized concept injection. However, synthesizing multiple customized concepts within a single image remains challenging due to the parameter pollution problem, where naive fusion leads to gradient conflicts and severe quality degradation. In this paper, we introduce ConWave-LoRA, a novel framework designed to achieve hierarchical disentanglement of object and style concepts in LoRAs. Supported by our empirical validation regarding frequency distribution in the latent space, we identify that object identities are predominantly encoded in high-frequency structural perturbations, while artistic styles manifest through low-frequency global layouts. Leveraging this insight, we propose a Discrete Wavelet Transform (DWT) based filtering strategy that projects these concepts into orthogonal optimization subspaces during contrastive learning, thereby isolating structural details from stylistic attributes. Extensive experiments, including expanded ablation studies on LoRA rank sensitivity and style consistency, demonstrate that ConWave-LoRA consistently outperforms strong baselines, producing high-fidelity images that successfully integrate multiple distinct concepts without interference.