Image-based approach to hand sign translation and implementation
Lucas Slomski, Michael Doyle, Ying Yu, Xin Shen · 2025
In this undergraduate research work, we focus on the process of sign language detection, recognition and translating it to standard English language using optical sensing, computer vision algorithms, pose detection software and artificial intelligence. Various methods of hand detection and recognition were studied and tested which include Mediapipe library under OpenCV and LSTM model. Upon completion of the translation process as well as implementation of the software, various parameters were tested. These included the accuracy of the object recognition and translation under different environments, the computation speed, and the performance of the hardware. In addition, the software was implemented into an embedded system to test the accessibility of communication with virtual home assistants. This research holds potential for integration with augmented reality technologies, which offers significant potential benefits to people with disabilities by enhancing both visual and hearing assistance through advanced devices.