Closed Domain Question-Answering Techniques in an Institutional Chatbot
Matthew Saad, Zakariya Qawaqneh · 2024
This paper introduces BrockportGPT, a specialized chatbot designed for SUNY Brockport that addresses the unique challenges of institutional question answering that general purpose Large Language Models (LLMs) face. Our approach leverages a combination of vanilla sequence-to-sequence modeling, fine tuning LLaMA-2, and Retrieval Augmented Generation (RAG). The methodology involves extensive data collection through web scraping, synthetic question generation, and a comprehensive look at information retrieval strategies, including the implementation of a question-topic classification system. Comparative analysis of the three approaches shows that finetuning and RAG have competitive performance, with RAG providing the most accurate and contextually relevant results, and finetuning having the superior dialog coherence. Through these results, BrockportGPT offers a model for developing an institutional chatbot, highlighting the potential for AI-driven education tools to improve information accessibility and dissemination processes.