Query-as-context Pre-training for Dense Passage Retrieval
Xing W, Guangyuan Ma, Wanhui Qian, Zijia Lin, Songlin Hu · 2023
Recently, methods have been developed to improve the performance of dense passage retrieval by using context-supervised pre-training.These methods simply consider two passages from the same document to be relevant, without taking into account the potential negative impacts of weakly correlated pairs.Thus, this paper proposes query-as-context pre-training, a simple yet effective pre-training technique to alleviate the issue.Query-as-context pretraining assumes that the query derived from a passage is more likely to be relevant to that passage and forms a passage-query pair.These passage-query pairs are then used in contrastive or generative context-supervised pre-training.The pre-trained models are evaluated on largescale passage retrieval benchmarks and out-ofdomain zero-shot benchmarks.Experimental results show that query-as-context pre-training brings considerable gains for retrieval performances, demonstrating its effectiveness and efficiency.