The Efficacy of an AI-Generated Questions on Medical and Nursing Student Knowledge Acquisition and Efficiency: A Learning Curve Analysis (Preprint)
Deepak Melwani, Riley M Silber, Andrew Asencio, M.A. Whitmill, Shan Rehman · 2025
BACKGROUND The integration of artificial intelligence (AI) with evidence-based academic principles like retrieval practice offers novel solutions to the challenges of knowledge development in health professions education. AI-driven platforms can deliver personalized active learning tools at scale, but empirical evidence of their effectiveness is needed. OBJECTIVE To evaluate the relationship between repeated engagement with dynamically generated AI questions and the mastery of specific educational topics by medical and nursing students. METHODS This study was a retrospective analysis of data from 506 medical and nursing students answering 93,568 practice questions. Data was anonymized and filtered for topics where students answered at least 6 sequential, unique, AI generated questions. The analysis focused on learning curves for accuracy (percent correct) and efficiency (median time to a correct answer) across the first six generated and answered practice questions on a given topic. Trends were also stratified by question difficulty (easy, medium, hard). RESULTS A strong, positive learning curve was observed, showing statistically significant improvements in both student accuracy (from 71.0% to 86.7%; p < 0.001) and cognitive efficiency (median time to a correct answer decreased from 41.9s to 17.9s; p < 0.001). The improvements followed a stepwise pattern, with the most significant gains occurring in the first three attempts. While students performed similarly across all question difficulties on their first attempt, a significant performance gap emerged in subsequent attempts. This powerful learning effect was observed across all levels of question difficulty (p < 0.001). CONCLUSIONS Repeated, targeted practice using an AI-powered question generation platform is strongly associated with improved learning outcomes and cognitive efficiency for medical and nursing students. These findings support the use of AI-driven educational systems to deliver effective retrieval practice that can be scaled to large cohorts and tailored to strengthen medical education for individual students.