LLM-empowered Dynamic Prompt Routing for Vision-Language Models Tuning under Long-Tailed Distributions
Yiwei Jia, Jiarui Ma, Xiangxian Li, Baiqiao Zhang, Xianhui Cao, Juan Liu, Yulong Bian · 2025
Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated impressive capability in visual tasks, but their fine-tuning often suffers from bias in class-imbalanced scenes.Recent works have introduced large language models (LLMs) to enhance VLM fine-tuning withsupplementaryy semantic information.However, they often overlook inherent class imbalance in VLMs' pre-training, which may lead to bias accumulation in downstream tasks.To address this problem, this paper proposes a Multi-dimensional Dynamic Prompt Routing (MDPR) framework.MDPR constructs a comprehensive knowledge base for classes, spanning multiple visual-semantic dimensions.During fine-tuning, the dynamic routing mechanism aligns global visual classes, retrieves optimal prompts, and balances finegrained semantics, yielding stable predictions through logits fusion.Extensive experiments on long-tailed benchmarks, including CIFAR-LT, ImageNet-LT, and Places-LT, demonstrate that MDPR achieves comparable results with current SOTA methods.Ablation studies further confirm the effectiveness of our semantic library for tail classes and show that our dynamic routing operates with a slight increase in computational overhead, making MDPR a flexible and efficient enhancement for VLM fine-tuning under data imbalance.The codes are available in https://github.com/Sha843/MDPR.