Sign Language to Text Translation with Computer Vision: Bridging the Communication Gap
So Xue Thong, Eng Lip Tan, Ching Pang Goh · 2024
This research paper addresses the communication barriers faced by individuals using sign language and those without hearing difficulties. With limited adoption of sign language and a scarcity of proficient human translators, there is a need for innovative solutions. This research presents a real-time sign language translation system using computer vision technology. The system utilizes Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks for static and dynamic sign recognition, respectively. Word segmentation which utilized ‘word-ninja’ and a Large Language Model (LLM) contributed to accurate sentence generation. The proposed system integrates machine translation and text-to-speech functionalities to improve the system's accessibility. The methodology involves data collection, landmark recognition, and the implementation of recognition and translation models. The results show impressive accuracy which are 99.20% and 90.08% for static and dynamic sign recognition respectively. However, issues such as environmental conditions have affected the detection accuracy and made errors in recognising similar signs. Sentence generation also provided a pretty decent result which is 90% of the accuracy from the output provided by the LLM model. Despite these challenges, the system still contributes to reducing communication barriers and promoting inclusivity in society.