Leveraging Prompt Learning for Robust Transfer in Long-tailed Distributions
Sunwon Jang, Suk‐Ju Kang · 2025
In real-world scenarios, training data often follow long-tailed distributions, where class imbalance hinders model performance, especially on underrepresented classes. While multi-modal foundation models like CLIP have shown strong generalization, existing adaptation methods have made limited use of text supervision, leading to biased accuracy and loss of pretrained knowledge. In this work, we propose a context learning-based approach to adapt CLIP to long-tailed settings. We introduce a hybrid strategy that combines class-specific and shared context, helping to preserve CLIP’s image-text alignment while improving performance across all class types. To further address class imbalance and maintain general knowledge, we incorporate tailored loss functions during training. Experimental results demonstrate that our method achieves higher overall accuracy and a smaller head–tail performance gap compared to existing fine-tuning approaches, highlighting the effectiveness of context learning for long-tailed adaptation of vision-language models.