Cyber Physical System with Real-Time Gesture Recognition for ISL Translation
A Afzal Pasha, Mohammed Ali Mohammed Al Sakkaf, S. Saleem, Nitin Rakesh, Hema Malini B H, H. S. Laxmisagar · 2025
Sign language is fundamental for deaf communities, yet existing translation tools often lack accessibility and scalability. To address challenges in existing solutions, this paper proposes a cyber physical system using muscle activity sEMG and motion IMU data to develop an embedded wearable device for interpreting gestures. Our system proposes the use of a hybrid CNN and LSTM model. A key contribution is the creation of a custom dataset for Indian Sign Language (ISL), as there is a scarcity of datasets combining sEMG and IMU signals for sign language recognition. Signals captured at 200 Hz processed and initial experiments on eight gestures achieved 87.3% classification accuracy, thus proving the system's viability. This work contributes to inclusive technology by providing an end-to-end Indian Sign Language (ISL) gesture-to-text or gesture-to-voice translation pipeline. Future efforts focus on improving continuous real time gesture recognition and expanding vocabulary to provide accessibility for individuals with hearing impairments.