LLMRAG: An Optimized Digital Support Service using LLM and Retrieval-Augmented Generation
Oscar Cederlund, Sadi A. Alawadi, Feras M. Awaysheh · 2024
This study investigates integrating large language models (LLMs) with retrieval-augmented generation (RAG) to automate instructional solution generation within a real-world IT service desk environment. The focus is on supporting technicians by suggesting solutions for incoming tickets. The research assesses the benefits and challenges of employing LLMs in digital support services by analyzing technician feedback and solution retention rates. A two-week controlled evaluation reveals that 38.4% of the generated solutions were retained by technicians, highlighting the significant potential of LLMs for enhancing support ticket resolution. While technician opinions regarding system usefulness varied, data indicates substantial engagement and the effectiveness of RAG in augmenting existing workflows. Key challenges identified include seamless system integration, handling unstructured data, and adapting the model to the specificities of the Swedish language context. Future research should investigate the impact on time efficiency and expand the evaluation sample size to strengthen the findings.