Recognizing Indonesian words based on visual cues of lip movement using deep learning
Griffani Megiyanto Rahmatullah, Shanq-Jang Ruan, Lieber Po‐Hung Li · Measurement · 2025
Lipreading is one of the techniques that can enhance speech perception. However, there are still limited studies of lipreading research focusing on low-resource languages, such as Indonesian. In this study, we introduce an instrument designed to generate lipreading datasets using CC BY video data available on YouTube called Lipreading Information Resource Assembler-Generator (LIRA-Gen). Using this instrument, we present the first Indonesian language lipreading dataset (IDLRW) containing over 48,000 videos with 100-word categories spoken by various persons in natural conditions. Also, we developed a deep learning architecture consisting of an Advanced Residual Network (ARN) using ResNet-34 incorporated with a Channel Spatial Attention (CSA) module, improved sequence modeling by fusing Bi-Gru with Mamba (BGM), an integrated word decision module, and fine-tuned hyperparameter. Our measurement shows that it reaches an accuracy of 60.51% on the IDLRW dataset and outperforms state-of-the-art lipreading models from another dataset even without implementing an additional learning strategy.