UbiLearn: Supporting English-as-a-Foreign-Language Learners in Reflecting on Conversations Using a Smartwatch MHCI033
Riku Arakawa, Manami Nakagawa, Hiromu Yakura · Proceedings of the ACM on Human-Computer Interaction · 2025
What new opportunities can the current ubiquitous computing and AI technologies provide to support English-as-a-Foreign-Language (EFL) learners? To answer the question, we began with a formative study with EFL learners, uncovering multiple challenges during conversations with others and their desire to review such scenes later. We implemented a smartwatch prototype, UbiLearn , which features hand gesture recognition for in-situ multi-context annotation to save moments when learners face difficulty. The annotation is used to generate personalized educational material powered by speech and natural language processing. Through a series of studies, we demonstrated the feasibility and preferred usability of UbiLearn, leading to learners’ enhanced learning satisfaction. Moreover, the annotation data promoted the role of instructors by enabling the tracking of learners’ in-situ proficiency outside their tutoring sessions. We conclude by highlighting emerging opportunities for learners enabled by mobile and AI technologies, along with key considerations.