An Intent-Filtered Retrieval-Augmented Generation Chatbot for University Admissions in Vietnam
Huy- Trung Nguyen, Quoc-Dung Ngo, Quang-Dung DANG · 2024
Nowadays, universities invest significant time and resources in enhancing customer service, information dissemination, teaching support, and admission. The traditional admission processes at universities face numerous challenges, especially the overload and the lengthy time required to communicate with the admissions office for timely responses. This study proposes an approach that applies an intent classification filter combined with Retrieval Augmented Generation (RAG) to develop a chatbot system to assist university admissions. We utilize a dataset of questions and answers and relevant documents to optimize the knowledge base. We experiment with the system on a dataset comprising information related to admissions at the Posts and Telecommunications Institute of Technology in Vietnam, including 102 documents with 243 pages and 1223 historical question-answer records for 25 intents. The results indicate that the proposed approach is intent-filtered RAG, which performs better than traditional RA G, RA G- Fusion, and even intent-based combined with them. We achieved a 96.59% context recall as evaluated by the RAGAS framework and a 93.52% accuracy as assessed manually on 386 test questions, improving by 11.65% to traditional RAG. This approach can leverage past knowledge and the latest updates over time by utilizing both documents and questions. Therefore, it can adapt to the evolving needs of universities each year and has the potential to scale to other universities.