Implementation of an AI English-Speaking Interactive Training System Using Multi-Model Neural Networks
Ching‐Ta Lu, Yen-Yu Lu, Yi-Ru Lu, Ying-Chen Pan, Yu-Chun Liu · IEEE Access · 2025
Many people can read and listen to English well but may need help to speak it well. This paper aims to implement an AI English-speaking interactive training (AIESIT) system based on AI for special-purpose English-speaking training research, enabling students to express and communicate in professional English naturally. We will provide new tools for solving the software design in English-speaking training. The proposed AIESIT system integrates generative AI, speech recognition, and body recognition. The AIESIT system uses generative AI to generate an AI agent with mouth shapes that match the English speech, which enhances the user’s feeling of being in the real world. The speech recognition system recognizes the voice content of the user’s response for passing the evaluation. Since the AIESIT system does not have a natural person online, users can speak English boldly to improve their oral skills. During the learning process, OpenPoseNet is used to recognize whether the user has poor posture or leaves the seat during the learning process. Eye CNN is used to recognize whether the user falls asleep during the learning process and as a reference for continuing the lesson. Finally, the learning trajectory, including the recognition rate and response time, is output to score the performance of the English dialogues. The results of multiple user tests have shown that increasing the number of practice sessions can improve the English speaking, encouraging users to keep practicing, and this system helps improve their English speaking.