Generative AI Agents in Language Learning: A Randomized Field Experiment
Gyeombi Cheon, Yunmin Choi, Dongwon Lee, Jiye Baek · Proceedings of the ... Annual Hawaii International Conference on System Sciences/Proceedings of the Annual Hawaii International Conference on System Sciences · 2025
Artificial Intelligence (AI) is revolutionizing education, particularly with advancements in generative AI conversational agents. Our study investigates the effectiveness of these generative AI agents in enhancing English-speaking skills compared to human agents. Through a randomized field experiment involving 363 participants, we found that unrestricted use of AI tools led to a 5.90% improvement in lexical diversity, as measured by the Type-Token Ratio (TTR), highlighting the benefits of self-paced learning. Notably, learners with below-average proficiency experienced a 9.53% improvement in TTR, suggesting AI's potential to bridge educational equity gaps. Moreover, AI tools significantly reduced evaluation apprehension, further enhancing learning outcomes. These findings underscore AI's capacity to provide personalized, anxiety-free learning environments, particularly for students with lower proficiency, and offer valuable insights for integrating AI into educational strategies to foster more inclusive learning experiences.