Enhanced Sign Language Translation Between American Sign Language and Indian Sign Language Using LLMs
Malay Kumar, S. Sarvajit Visagan, Tanish Sarang Mahajan, Anisha Natarajan, P S Sreeja · IEEE Access · 2025
This research introduces a comprehensive framework designed to bridge the communication gap between American Sign Language (ASL) and Indian Sign Language (ISL). The proposed system employs a hybrid deep learning model for ASL gesture recognition, integrating a random forest classifier (RFC) and a convolutional neural network (CNN) to enhance accuracy. Recognized gestures are converted into text, which is refined using a fine-tuned Large Language Model (LLM) for contextual and grammatical accuracy. The corrected text is then synthesized into ISL gestures using RIFE-Net, a real-time intermediate flow estimation network, to generate smooth and natural gesture videos. The framework addresses key challenges such as gesture variability and linguistic differences between ASL and ISL, achieving a translation accuracy of 93.0% for gesture recognition and 94.2% for text correction. Initial experimental results demonstrate real-time processing capabilities, averaging one gesture per second, with video outputs at 60 FPS. This system not only facilitates seamless communication between ASL and ISL users but also lays the groundwork for scalability to other sign language pairs. The results highlight the potential to improve accessibility and inclusion of the global deaf community, paving the way for future research in multimodal sign language translation systems.