A Recurrent Neural Network Approach to Image Captioning in Braille for Blind-Deaf People
Sameia Zaman, Mohammed Abid Abrar, Mohammad Muntasir Hassan, A N M Nafiul Islam · 2019
Limitation of resources in braille is one of the biggest obstacles faced by the blind community trying to learn and integrate themselves better in our societies. The fact that not only text but also images are used for communicating information does not help their cause. In this paper, we intend to help not only the visually impaired but also people with deaf-blindness, by converting images into captions which then can be read in braille. Being able to automatically translate the content of an image using properly formed English sentences is a challenging task in itself, but it could have great impact by helping a blind person better understand his/her surroundings and even in mundane tasks like browsing the web for example. While text-to-speech methods can be then implemented using voice synthesizers for blind only people, the text needs to be converted into braille for people who have both hearing and sight loss. Our paper has been centered on the concept of transforming images into grade 1 braille. Here we present a deep recurrent neural network architecture that automatically generates brief descriptions of images directly in braille. Our model achieves a BLEU-4 score of 0.24 on the Flickr 8K Dataset, which is comparable to current text based state-of-the-art deep learning models. Moreover, the generated captions are translated into haptic feedbacks readable for the blind by a microcontroller-based system on a 2 × 3 arrangement of push-pull solenoids.