Indian sign language to voice using ESP32_Cam for hand tracking and MediaPipe for hand gesture detection
Shubham Sony, Arjun Lande, Ankur Banerjee, Ajay Kumar Sharma, Shamneesh Sharma · 2025
Communication gaps experienced by people with hearing and speech disabilities continue to be a major issue in the modern world, particularly when communicating with people who are not familiar with sign language to bridge this gap, this study introduces a real-time Indian Sign Language (ISL) to Voice translation system based on an ESP32-CAM module for capturing hand gestures and Media-Pipe for precise hand landmark detection and gesture recognition. The system identifies ISL alphabets and numbers using effective hand tracking and gesture segmentation, providing smooth conversion of sign language to text as well as synthesized speech output. Media-Pipe's hand tracking improves the precision of gesture recognition by capturing 21 critical hand landmarks while a light machine learning model determines the gestures. The identified text is then converted to voice with the help of text-to-speech (TTS) synthesis, enabling clear communication for the hearing and speech-impaired. Differing from conventional systems depending on heavy hardware or being confined to English speech, this project focuses on an embedded, cost-effective solution via ESP32-CAM, thereby being portable. In addition, the flexibility in the system will enable future enhancement for multilingual, even regional languages such as Malayalam, to extend inclusivity. This paper describes the architecture, implementation, and experimental performance of the system, showing its efficiency and accuracy in real-time ISL gesture recognition and voice generation. The system proposed here has great potential in filling the communication gaps, enabling independence and social inclusion of hearing and speech disabled individuals.