CHDNet: Enhanced Arbitrary Style Transfer via Condition Harmony DiffusionNet
Wenkai He, Jianhui Zhao, Ying Ze Fang · 2024
Arbitrary Style Transfer (AST) renders an image by adopting the style of any chosen artwork while preserving its content structure. Despite the widespread popularity of feedforward AST methods, they tend to merely optimize the statistical characteristics of images, leading to unnatural outputs and displeasing low-quality distortions. In contrast, diffusion models effectively address this issue by reconstructing the overall image. Unlike typical image generation tasks controlled by a single condition, style transfer demands the simultaneous consideration of multiple conditions. We propose Condition Harmony DiffusionNet (CHDNet), which distinguishes between style and content conditions, integrating them into harmonious conditions to collaboratively guide the generation process. We innovatively introduce a content aware attention, designed to extract semantic features of the content image across multiple dimensions, distinctly setting it apart from the style condition. Furthermore, we have improved the skip connections in the diffusion model, which introduces a slight increase in model complexity but results in a substantial improvement in the representation of fine details. Further refinements in the image sampling process empower us with great control over the stylization effect in the generated results. Our method successfully employs diffusion models via harmonious conditions to solve AST, achieving outstanding effects. Experiments demonstrate that our method achieves stateof-the-art arbitrary style transfer.