Optimization of Customer Feedback Summarization Using Large Language Models (LLM) and Advanced Retrieval-Augmented Generation
Buchepalli Praneeth, Mohana, Eshitha Chowdary Nattem, Kamala Jetti, B K Kavyashree, Dilshan Rakshitha, P. Ramakanth Kumar, K. Sreelakshmi · IEEE Access · 2025
Customer feedback, often shared through online reviews, plays a crucial role in shaping business strategies. However, the overwhelming volume of such reviews poses two major challenges: valuable insights often go unnoticed, and manual analysis introduces human bias. To address this, we propose a system that leverages large language models (LLMs) integrated with the LangChain framework to answer natural language queries over customer reviews. A synthetic dataset was created to resemble food delivery reviews typically seen on the Play Store and was stored in a vector database. On receiving a user query, relevant reviews are retrieved using advanced Retrieval-Augmented Generation (RAG) techniques, namely Hierarchical Chunk Retrieval and RAG Fusion that are further refined using the Declarative Self-improving Python (DSPy) framework to generate accurate, grounded responses. The system was evaluated using LLaMA-3-8B-InstructLite, GPT-3.5-Turbo, and Gemini-1.5-Pro, and compared against two non-LLM baselines: BM25 and a fine-tuned BERT model. Results show that our LLM-based pipeline outperforms by 15% in semantic and factual accuracy. Component-level analysis showed that enhanced retrieval strategies improved semantic relevance by up to 4.9%, lexical coverage by 12.1%, and factual consistency by 9.9% over traditional RAG. Further integration of DSPy led to an additional 10.8% boost in linguistic fluency and a 9.0% gain in factual alignment. Among the evaluated models, Gemini-1.5-Pro combined with RAG Fusion and DSPy produced the most fluent and factually accurate responses, demonstrating the effectiveness of combining hybrid retrieval with LLM-driven reasoning for query-based feedback response system.