CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking
George Zerveas, Navid Rekabsaz, Daniel Cohen, Carsten Eickhoff · 2022
Contrastive learning has been the dominant approach to training dense retrieval models.In this work, we investigate the impact of ranking context -an often overlooked aspect of learning dense retrieval models.In particular, we examine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of random negatives, and a fully list-wise loss.To incorporate these factors into training, we introduce Contextual Document Embedding Reranking (CODER), a highly efficient retrieval framework.When reranking, it incurs only a negligible computational overhead on top of a firststage method at run time (∼ 5 ms delay per query), allowing it to be easily combined with any state-of-the-art dual encoder method.Models trained through CODER can also be used as stand-alone retrievers.Evaluating CODER in a large set of experiments on the MS MARCO and TripClick collections, we show that the contextual reranking of precomputed document embeddings leads to a significant improvement in retrieval performance.This improvement becomes even more pronounced when more relevance information per query is available, shown in the TripClick collection, where we establish new state-of-the-art results by a large margin.