NuevAI: Streamlined Dataset Generator and Human Evaluator System for Development of Pedagogical Conversational Agents
Salman Sayeed, Saad Mohammad Rafid Pial, Anindya Iqbal · 2025
The development of Pedagogical Conversational Agents (PCAs) made significant progress due to the rise of Large Language Models (LLMs).Yet, creating and evaluating PCAs remains a technically challenging task for educators.Our work presents a semiautonomous system, NuevAI; designed to democratize the access to AI-enhanced educational tools by simplifying the development of PCAs.NuevAI consists of two complementary platforms that help educators create structured, multi-turn conversational datasets from various data sources for fine-tuning LLMs and also facilitates systematic assessment of PCA conversations by learners.From the evaluation of 16 educators from STEM subjects, our system shows notable improvements compared to traditional methods: 74.5% usability score (vs.50.9% baseline), 81.2% task efficiency (vs.49.1%), and 80.8% time efficiency (vs.32.5% baseline).The system achieved a Net Promoter Score of 68.8% (vs.-62.5% baseline) and reduced cognitive workload by 40% as measured by NASA-TLX assessment.By providing user-friendly and intuitive interfaces and automated workflows, NuevAI empowers educators to create adaptive learning tools through PCAs, aligned with specific pedagogical objectives without requiring technical expertise in the field of AI.