A Novel Sentence‐Level Visual Speech Recognition System for Vietnamese Language Using ResNet3D and Zipformer
Phat Nguyen Huu, Thach Ho Sy · Modelling and Simulation in Engineering · 2025
This paper presents the first sentence‐level visual speech recognition (VSR) system specifically designed for the Vietnamese language. We have developed a unique dataset comprising 115 h of video recordings from over 100 speakers, focusing on single‐speaker scenarios. The proposed VSR system utilizes a ResNet3D architecture as the visual frontend, paired with a neural transducer framework featuring a Zipformer speech encoder. It incorporates a stateless decoder that considers two preceding tokens and is optimized with a pruned‐RNNT loss function. Experimental results show that our system achieves a word error rate (WER) of 27.14% and a character error rate (CER) of 20.45% on single‐speaker tasks, demonstrating significant progress in VSR for Vietnamese.