Modifying Generative Distributions in Latent Diffusion Models to Improve Alignment with Desired Properties
Sven Sickert, Maria Gogolev, Niklas Penzel, Tim Büchner, Joachim Denzler · 2025
Models like DALL-E and Stable Diffusion substantially improved image generation. While visually convincing, they still exhibit distinct differences compared to real-world images and desired image distributions. Previous studies already identified issues at a spectral level. We also find discrepancies in terms of style authenticity and aesthetics. Based on these insights we investigate three distinct strategies to modify such models and enhance their alignment for selected image distributions of artworks. First, prompt optimization can be done without updating the model, but has limited efficacy. Fine-tuning the U-Net component can enhance denoising capabilities, leading to improved image quality. Finally, modifying the image decoder can help correct spectral misalignments. Our experiments on the ArtEmis dataset using three complementary measures show that high-frequency artifacts remain challenging, but alignment can usually be improved by at least 10%.