Gesture talks real-time sign language recognition and animation system using AI
Kamalesh Thoppaen Suresh Babu, Srikanth Banoth, Vijaya Lakshmi Muvvala, Mohammad Shafee, Shravan Kumar Ainala · World Journal of Advanced Engineering Technology and Sciences · 2025
Effective communication between deaf and hearing individuals remains a significant challenge due to the fundamental differences in language modalities. GestureTalk presents a real-time, AI-driven communication system designed to bridge this gap by enabling seamless bidirectional interaction. The proposed solution leverages state-of-the-art technologies including Automatic Speech Recognition (ASR), Natural Language Processing (NLP), gesture detection using YOLO, and pose estimation with DWpose and MediaPipe. Spoken language is transcribed and translated into American Sign Language (ASL) gloss, then rendered as realistic 3D animated sign language via a virtual avatar. In the reverse direction, the system captures and interprets sign language gestures in real time, converting them into textual output for hearing users. Designed with real-time performance, high accuracy, and user accessibility in mind, GestureTalk serves as an inclusive interface for communication particularly suited for digital contexts such as video conferencing. This system offers a scalable and adaptable solution, contributing meaningfully to assistive technology and digital accessibility for the deaf and hard-of-hearing communities.