DynaStyle: Mitigating Content Leakage by Dynamic Layer Routing in Stylized Image Generation
Sheng Shi, Bing Hao, Peng Wang, Jianping Fan · ACM Transactions on Multimedia Computing Communications and Applications · 2026
Diffusion models have become a dominant framework for image generation, demonstrating greater stability and performance compared to generative adversarial networks (GANs). Recent developments in text-to-image diffusion models have significantly enhanced capabilities in style transfer. However, existing adapter-based approaches face significant challenges in disentangling content and style representations. We observe that current methods, such as IP-Adapter, are prone to substantial content leakage, wherein features from the style image interfere with the integrity of the target content, leading to semantic distortions and degradation in output quality. This issue arises primarily from fixed-weight cross-attention mechanisms that fail to account for the hierarchical structure of the diffusion process. To address this limitation, we propose DynaStyle, a novel dynamic routing architecture that enables layer-adaptive conditioning through learned attention reweighting. The proposed framework incorporates a dynamic gating mechanism with negligible parameter overhead (0.5M per model) to modulate the influence of image prompts across transformer layers, facilitating stage-aware feature fusion during the denoising stages. DynaStyle jointly optimizes for both style fidelity and content preservation within an end-to-end trainable framework, while retaining compatibility with custom models derived from the same base diffusion model. Extensive experiments across diverse content and style domains demonstrate that DynaStyle achieves state-of-the-art performance, particularly in preserving semantic content and ensuring stylistic coherence.