Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without Homomorphic Encryption
Po-Chu Hsu, Zejun Yu, Shuhei Mise, Hideaki Miyaji · 2025
Text-to-image generation is trending in the generative AI field. Stable Diffusion is the state-of-the-art among open-source projects. Many artists and service providers customize the diffusion model for special textures. However, there is no protection for the privacy of the user's input text prompt, output image, and the customized model on the server. Privacy is crucial for user trust and protecting intellectual property. Existing privacy-preserving diffusion models use fully homomorphic encryption (FHE), which is time-consuming and can degrade image quality. We propose Privacy-Diffusion, a framework that preserves privacy without FHE by leveraging the irreversible properties of neural network layers and the property that in the diffusion process, the predicted noise is a normalized Gaussian distribution. Our framework protects clients' input text prompts and generated images from the server and safeguards customized models from clients. Compared with existing research HE-diffusion which spent 200% extra time and visible quality loss, our protocol can reach the same security level with only 4% extra time and has no quality loss. To our knowledge, we are the first to achieve this goal without FHE while maintaining high-quality image output.