Teaching Iranian Sign Language via a Virtual Reality-Based Game
Amirali Mazhari, Parsa Esfandiari, Alireza Taheri · 2022 10th RSI International Conference on Robotics and Mechatronics (ICRoM) · 2022
Nowadays, a considerable number of people have hearing disabilities. These people have difficulties communicating with a major part of society. This paper’s main goal is to create an attractive environment to teach Iranian Sign Language (ISL) to children to increase the number of people who can use sign language in the future. In order to achieve this purpose, a virtual reality game has been designed with the Unity game engine that the user plays on Oculus Quest 2, using eight ISL signs. These signs are detected by a webcam and processed in real time by a deep learning algorithm. After recognizing the performed sign, the command will be run in the game. The used deep learning algorithm includes two Deep Neural Network models. One is based on Convolutional Neural Network (CNN) and the other one is based on Long Short-Term Memory (LSTM) with test accuracies of 99.7% and 99.1%, respectively. The game has been tested on a number of participants. After analyzing their recorded data during the tests, it is indicated that considerable improvements have been occurred on their performances. It is observed that players could play 195.36% longer in the second half of their tries compared to the first half.