Evaluating and Enhancing RAG Systems through Test and Source Analysis
Kazunori Matsumoto, Zealan Shi, Oranus Kotsuwan · IIAI Letters on Informatics and Interdisciplinary Research · 2025
This paper presents a prototype Retrieval-Augmented Generation (RAG) system developed for university curriculum guides and evaluates its performance through experiments. RAG, which combines large language models (LLMs) with independent information sources, is emerging as a solution to address generative AI challenges such as hallucinations and the lack of domain-specific knowledge. By prioritizing information from dedicated databases, RAG can enhance factual accuracy and reduce hallucinations. Through experimental trials, the system demonstrated reliable performance in some cases, although issues related to the quality of information sources and data extraction were identified. These findings underscore the importance of robust testing and systematic revisions of information sources. This paper reports on an outline of the system implementation, the guides for improvement, and the experimental results. We find that an iterative improvement process is crucial for enhancing the overall quality of RAG. This process involves not only optimizing retrieval and generation mechanisms but also continuously reviewing and refining the information sources themselves, the system can systematically adapt to ensure sustained relevance and improved response accuracy over time.