Building Intelligent Academic Service Agents: FAQ Extraction and Chatbots With Large Language Models
Chatchai Wangwiwattana, Worawut Jantarick · 2024
This study proposes an academic chatbot system utilizing large language models (LLMs) to assist students with university-related tasks. The system is designed to simplify the maintenance of school information by extracting FAQs directly from instant messaging platforms. To ensure safety and reliability, the system incorporates a two-step filtering process. In this study, 300 real-world conversations were analyzed, resulting in the extraction of 95 registrar-related Q&A pairs that were subsequently trained into the system. The system’s performance was evaluated based on three key metrics: context relevance, answer relevance, and groundedness. The results demonstrated high satisfaction levels, with scores of 0.94, 0.93, and 0.92, respectively.