DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
Xueguang Ma, Xi Victoria Lin, Barlas Oğuz, Jimmy Lin, Wen-tau Yih, Xilun Chen · 2025
Large language models (LLMs) have demonstrated strong effectiveness and robustness when fine-tuned as dense retrievers.However, their large parameter size presents significant computational challenges at inference time.While smaller retrievers offer better efficiency, they often fail to generalize effectively with limited supervised fine-tuning data.In this work, we introduce DRAMA, a training framework that leverages LLMs to train smaller generalizable dense retrievers.In particular, we adopt pruned LLMs as the backbone and train on diverse LLM-augmented data in a single-stage contrastive learning setup.Experiments show that DRAMA offers better multilingual and long-context capabilities than traditional encoder-based retrievers, and achieves strong performance across multiple tasks and languages.1 * Equal contribution.† Work done while at Meta. 1 Code and checkpoints will be available at https://github. com/facebookresearch/dpr-scale/tree/main/drama.