Assessing Large Language Models as Agile Scrum Masters: A Comparative Study of Project Planning Efficiency
Amirali Shahriary, Mohammadsaeid Sedighi, Nima Tajik, Mohammadali Shahinfar, Amirhossein Rahati Asiyabar · 2025
Agile project management has become a cornerstone of modern software development, with Scrum Masters playing a critical role in ensuring project success. The advent of large language models (LLMs) has introduced new possibilities for automating project planning tasks, raising questions about their effectiveness compared to human expertise. This study aims to evaluate the feasibility of LLMs in Agile project planning by comparing their performance against human Scrum Masters. A standardized project planning template was used to ensure uniformity across all generated plans, focusing on key Scrum principles such as task breakdown, sprint organization, and risk management. The project plans produced by both LLMs and human Scrum Masters were assessed by software engineering faculty members from the University of Tehran based on predefined evaluation criteria. The results revealed that certain LLMs, including ChatGPT and Gemini Flash 1.5, outperformed human Scrum Masters in terms of operational feasibility, task clarity, and sprint organization. However, the findings also highlighted significant variability in the effectiveness of different models, emphasizing the critical role of prompt engineering in optimizing output quality. While LLMs demonstrated their potential to enhance efficiency and scalability in structured environments, they lacked the human-centric qualities necessary for dynamic project adaptation, risk identification, and team management. This study concludes that LLMs can serve as valuable augmentation tools for Agile project management, complementing human expertise rather than replacing it. Future works should focus on integrating LLMs into dynamic, real-world Agile environments and exploring hybrid approaches that leverage both AI capabilities and human intuition.