Learner autonomy and mode selection in dual-mode chatbot: Students' interaction patterns and learning outcomes in an online course
Songhee Han · Computers & Education · 2025
While educational chatbots are increasingly used in online courses, most research has focused on single-mode chatbots, with little attention to how students engage with multi-mode chatbots. Limited empirical studies have examined how students exercise autonomy when choosing between a general large language model (LLM)-based mode and a reference-based mode powered by retrieval-augmented generation (RAG). This study examines how students in a large-scale online course engaged with a dual-mode chatbot that allowed them to choose between general (LLM-based) and reference-based modes (LLM-RAG). Grounded in a conceptual framework linking learner autonomy, user factors, and learning outcomes, the study employed a mixed-methods explanatory sequential design using chatbot interaction logs, learning management system logs, server data, and survey responses. Findings showed that 41.3% of the participants were unaware of or indifferent to using the mode selection feature despite each interaction beginning with a system-generated message explaining each mode’s functionality. Among those aware, general mode users prioritized convenience and open-ended use, while reference-based mode users emphasized course-aligned assistance and demonstrated greater hyperlink engagement. Hyperlink use was positively correlated with social presence, teaching and cognitive presence, self-regulation, and perceived ease of use. Course completion showed a modest positive correlation with perceived ease of use but not with other variables. Interaction behaviors differed between the two chatbot modes, with exploratory and socio-emotional talk more prevalent in the general mode, and course-related sentence questions dominating the reference-based mode. These findings highlight the need for intentional design that fosters learner autonomy and aligns chatbot functionality with instructional goals. • Students chose between LLM and LLM-RAG chatbot modes in a real online course. • Over 40% were unaware of the chatbot mode selection option. • LLM mode was used for convenience; LLM-RAG for course-specific support. • LLM-RAG mode use is linked to stronger perceived learning experiences. • Informed autonomy is key to effective chatbot integration in online learning.