Creating an Accessible Future: Developing a Sign Language to Speech Translation Mobile Application with MediaPipe Hands Technology
Melchizedek Recalde, Andre Losito, Manuel James Morillo, Ciandra Pancho, James Reyes, Jenelyn Aranas · 2023
The SignTech mobile application utilizes MediaPipe Hands technology to convert hand gestures of the American Sign Language (ASL) alphabet into text and subsequently into speech. Through MediaPipe Hands Solutions Framework, the developers mapped and calculated the hand landmarks to form each ASL letter. They compared the calculated hand landmarks to the landmarks defined for each ASL letter and used a matching algorithm to determine the best match. This innovative solution breaks down communication barriers, allowing deaf individuals to express themselves more easily and enabling hearing individuals to understand and engage with sign language users. The developers conducted a user study to evaluate the effectiveness and usability of SignTech, and the results demonstrate its success in translating sign language into speech. This research provides valuable insights into the development of technology for sign language translation and highlights the potential of MediaPipe Hands technology in improving communication between deaf and hearing individuals with a mobile application.