A Tone-Assisted Chinese Silent Speech Input Method for Virtual Reality
Xinyu Zhang, Xinyuan Shan, Yang Zhou, Zihong Chen, Peng Tan, Youlu Xing, Luo Bin, Zhigeng Pan, Guihuan Feng · International Journal of Human-Computer Interaction · 2025
Silent speech input offers a fast alternative to other text entry methods in VR, such as virtual keyboard. However, in Chinese, the lack of tone creates ambiguity, as similar lip movements can convey different meanings depending on the tone. We proposed a tone-assisted silent speech input method that improves accuracy. We first explored five tone-input methods in VR, demonstrating that tone input can be synchronized with silent speech. Next, we implemented the most preferred tone-input method and designed a corresponding multimodal lip-reading model, ToneLipNet. ToneLipNet showed significantly higher accuracy than the baseline model without tone. Then, we conducted a five-day study to investigate the long-term potential of our input method. Participants quickly adapted, achieving high input speed and good accuracy. Finally, we conducted a user study to investigate how users coordinate silent speech input and tone input in VR.