A Two-Stage Adaptation of Large Language Models for Text Ranking
Longhui Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, Meishan Zhang, Min Zhang · 2024
Text ranking is a critical task in information retrieval.Recent advances in pre-trained language models (PLMs), especially large language models (LLMs), present new opportunities for applying them to text ranking.While supervised fine-tuning (SFT) with ranking data has been widely explored to better align PLMs with text ranking goals, previous studies have focused primarily on encoder-only and encoder-decoder PLMs.Research on leveraging decoder-only LLMs for text ranking remains scarce.An exception to this is Ran-kLLaMA (Ma et al., 2023a), which uses direct SFT to explore LLaMA's potential for text ranking.In this work, we propose a two-stage progressive paradigm to better adapt LLMs to text ranking.First, we conduct continual pretraining (CPT) of LLMs on a large weaklysupervised corpus.Second, we perform SFT, and propose an improved optimization strategy building upon RankLLaMA.Our experimental results on multiple benchmarks show that our approach outperforms previous methods in both in-domain and out-domain scenarios.* Corresponding author. 1 As the candidate set is usually small, while the first step focuses on efficiently collecting candidate documents, text ranking tends to prioritize performance over efficiency.The farad (symbol: F) is the SI derived unit of electrical capacitance, ...