Indian Sign Language Generation – A multi-modal approach
Pooja Chaudhari, Mangesh V. Bedekar · 2023
As social beings, communication is an important skill for us humans. But some people who have hearing issues face problems while communicating in their day to day lives. They use Sign Language as a way to express themselves with the help of hand gestures and facial expressions. In today’s times when we have the whole world on our palms, online content in multiple forms is a large source of entertainment and infotainment for us. The accessibility of online content is quite high, but for people with hearing loss, it is still not accessible enough. In this research, we propose a method of generating Indian Sign Language gestures from text, subtitles of video content, audio and image with text as inputs. For the video content we have taken Youtube videos as a case study. After further processing and with help of SiGML files, we get the animated gestures as output. Another method we propose is for recognizing the ISL gestures with the help of spatial and temporal feature extraction using Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). These will give the word for the ISL gesture being played. We used accuracy and confusion matrix as performance metrics for this methodology. We recorded an accuracy of 96% in ISL Recognition methodology.