Condenser: a Pre-training Architecture for Dense Retrieval
Luyu Gao, Jamie Callan · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
Pre-trained Transformer language models (LM) have become go-to text representation encoders.Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval.However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations.This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation.We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation.Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks. 1