LADDER: LLM-Annotated Data for Dogfooded Evaluation of Rankings
Mattia Ottoborgo · 2025
In this paper we showcase the implementation of LADDER: A method utilizing Large Language Model to annotate thousands of consumer reviews to train a point-wise learning to rank algorithm.By applying LADDER, we significantly improved the relevance of the top 4 reviews presented to users, demonstrably reducing the need to access the full review collection by 5%.This outcome highlights LADDER's ability to enhance user experience by providing sufficient information within the initial review set, thereby streamlining the decision-making process.We discuss the efficiency gains in large-scale data labeling, the positive impact on trust and relevance in review presentation without sacrificing usability, and key insights into effectively integrating domain expertise into LLM annotation for high-quality results.