COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust Learning
Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, Arnold Overwijk · 2022
We present a new zero-shot dense retrieval (Ze-roDR) method, COCO-DR, to improve the generalization ability of dense retrieval by combating the distribution shifts between source training tasks and target scenarios.To mitigate the impact of document differences, COCO-DR continues pretraining the language model on the target corpora to adapt the model to target distributions via COtinuous COtrastive learning.To prepare for unseen target queries, COCO-DR leverages implicit Distributionally Robust Optimization (iDRO) to reweight samples from different source query clusters for improving model robustness over rare queries during fine-tuning.COCO-DR achieves superior average performance on BEIR, the zero-shot retrieval benchmark.At BERT Base scale, COCO-DR Base outperforms other ZeroDR models with 60× larger size.At BERT Large scale, COCO-DR Large outperforms the giant GPT-3 embedding model which has 500× more parameters.Our analysis show the correlation of COCO-DR's effectiveness in combating distribution shifts and improving zero-shot accuracy.Our code and model can be found at https://github.com/OpenMatch/COCO-DR.