Contrastive Learning to Improve Retrieval for Real-World Fact Checking
Aniruddh Sriram, Fangyuan Xu, Eunsol Choi, Greg Durrett · 2024
Recent work on fact-checking addresses a realistic setting where models incorporate evidence retrieved from the web to decide the veracity of claims.A bottleneck in this pipeline is in retrieving relevant evidence: traditional methods may surface documents directly related to a claim, but fact-checking complex claims requires more inferences.For instance, a document about how a vaccine was developed is relevant to addressing claims about what it might contain, even if it does not address them directly.We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for this setting.By leveraging the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents, we fine-tune Contriever with a contrastive objective based on multiple training signals, including distillation from GPT-4, evaluating subquestion answers, and gold labels in the dataset.We evaluate our model on both retrieval and end-to-end veracity judgments about claims.On the AVeriTeC dataset, we find a 6% improvement in veracity classification accuracy.We also show our gains can be transferred to FEVER, ClaimDecomp, HotpotQA, and a synthetic dataset requiring retrievers to make inferences.