In-Context Learning in LLMs to Improve Retrieval Models
Nilanjan Sinhababu · 2024
A supervised ranking model, although generally more effective than traditional approaches, often requires intricate processing that involves several stages.This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that can work in a zero-shot manner.Current zero-shot re-rankers demonstrate promising results, achieving effective performance without the need for training data and operating with streamlined pipelines.However, since zero-shot inference relies only on preexisting knowledge and generalization of the model and operates without access to a task-specific training set, its performance is typically less robust than that of supervised models.This tutorial covers a technique that improves the zero-shot ranking performance of LLMs using few-shot in-context learning.This technique requires a similar query and a pair of documents as an in-context example, which provides context for the query and definition of the downstream task.Providing these localized in-context examples is effective while being a non-parametric way of controlling the LLM ranking predictions.