The blueprint of a new fact-checking system: A methodology to enrich RAG systems with new generated datasets
Salvador López-Joya, Jose A. Diaz-Garcia, Maria Dolores Ruiz, Marı́a J. Martı́n-Bautista · Computers & Electrical Engineering · 2025
In an era where digital misinformation spreads rapidly, Artificial Intelligence (AI) has become a crucial tool for fact-checking. However, the effectiveness of AI in this domain is often limited by the availability of high-quality and scalable datasets to train and guide algorithms. In this paper, we introduce VERIFAID (VERIfication FAISS-based framework for fake news Detection), a novel framework that improves fact-checking through a Retrieval-Augmented Generation (RAG) system based on automatically generated and dynamically growing datasets. Our approach improves evidence retrieval by building a scalable knowledge base, reducing the reliance on manually annotated data. The system consists of three key modules: two dedicated to dataset creation and one inference module that integrates advanced language models, such as LLaMA, within the RAG paradigm. To validate our methodology, we provide technical specifications for both the system and the dataset, together with comprehensive evaluations in zero-shot fact-checking scenarios. The results demonstrate the efficiency and adaptability of our approach and its potential to improve AI-driven fact verification at scale. • A RAG-based approach can compete with and outperform other fact-checking methods. • An auto-updating database enables real-time use and reduces human intervention. • Adding metadata and embedding distance improves classification in LLM contexts. • LLM behaviour improves with role specification and the inclusion of more chunks. • LLMs effectively extract claims and verdicts when data is in natural language.