An Efficient Speech to Sign Language Conversion and Text Recognition through Live Gesture
M. Kowsigan, Rahul Dhawan, Ankan Kundu · 2024
Despite advancements in technology, a significant portion of the global population (over 5%) continues to face communication barriers due to deafness and speech impairments. Existing solutions often lack the comprehensiveness and inclusivity required to address the diverse needs of this community. Traditional methods, relying solely on speech-to-text or text-to-sign language conversion, fail to facilitate seamless two-way communication, limiting interpersonal connections and fostering a sense of isolation. Our research proposes a novel, multimodal approach integrating speech-to-sign language conversion and live gesture-to-text recognition, leveraging audio and visual inputs. By combining these modalities, our system aims to bridge the communication gap, enabling real-time, bidirectional interactions between individuals with and without hearing or speech impairments. Employing speech recognition, natural language processing techniques, and machine learning models developed using Python, our solution incorporates a custom dataset, encompassing personalized hand symbols and offering the flexibility to choose between American Sign Language (ASL) and Indian Sign Language (ISL), promoting inclusivity and catering to cultural preferences. The necessity of this research lies in its potential to revolutionize assistive technologies, fostering a more connected and empathetic society. By providing a comprehensive platform that facilitates effective communication and mutual understanding, our solution improves quality of life, educational opportunities, and social inclusion for individuals with impairments.