Adapting General Pre-Trained Models for Medical Pathology Image Generation with Limited Data
Rui Li, Wenhao Wang, Baorong Liu, Jiayu Yu · 2025
Recent advances in text-to-image generation models have demonstrated impressive capabilities, however, these models often struggle to capture the subtle and complex characteristics of medical images, limiting their applicability in the medical domain. Training such models from scratch requires large computational resources and large-scale, annotated medical datasets which are often unavailable due to data scarcity and privacy constraints. In this study, we explore the use of Low-Rank Adaptation (LoRA) to fine-tune a pre-trained Stable Diffusion model, originally trained on natural images, for medical imaging tasks. Specifically, we finetune the model using only 100 real images per pathology from the CheXpert dataset on a single consumer-grade GPU. We evaluate the quality of the generated images using quantitative metrics and further assess their utility by augmenting classification datasets with the synthetic images. Our experiments demonstrate that LoRA fine-tuning can be completed within a few hours and yields high-quality, diagnostically relevant medical images. Moreover, incorporating synthetic images improves the performance of classification models, validating the practical utility of this approach in downstream tasks. The resulting LoRA parameter files are under 200 MB, making them easily shareable and alleviating privacy concerns. This work presents an efficient and accessible solution for medical image generation in resourceconstrained research environments.