Optimization Method for Controllable Image Style Transfer Based on DDPM Guided by Cross-Modal Text
Shengnan Ding, Yufeng Shi · IEEE Access · 2025
Traditional image style transfer methods rely mostly on a single image modal. Although they can introduce style features, it is difficult to achieve high-level semantic consistency while maintaining the content structure.The existing diffusion model-based schemes have problems such as insufficient cross-modal constraints and rough feature injection, resulting in insufficient semantic expression of the generated results and weak style expression.Therefore, this paper proposes a cross-modal image style transfer optimization method based on diffusion model. By introducing text description of content images as cross-modal guidance, CLIP extracts text features injected in the diffusion process, enhancing the maintenance and constraints of the content semantics by the generation results.At the same time, a progressive KV-blend style injection mechanism is designed to integrate the key-value features of the style image layer by layer in the multi-layer attention structure of diffusion inversion, making the style transfer more natural and clearer.Experimental results show that while maintaining the integrity of the content structure, the generated images are better than the existing methods in terms of semantic consistency, visual quality and style presentation.