English to Hindi Multi Modal Image Caption Translation
Jagroop Kaur, Gurpreet Singh Josan · Journal of scientific research · 2020
When it comes to describe image by words, the possibilities are numerous.Generating caption automatically from image is a well researched area.The concept is now advanced to utilise multiple models for translating captions in another language.One of the major hurdle in machine translation is word sense disambiguation.Visual cues from image can be helpful in disambiguate source words.System are being developed to generate image descriptions in a target language from an input image and one or more descriptions in source language.This paper describes the multi modal Neural Machine Translation systems for translating image captions from English to Hindi.Various multi-modal architectures were explored using local visual feature, global visual features, attention mechanisms, and pre-trained embedding.The features obtained form various models are integrated in various ways.We also tried re-ranking method.The systems are evaluated on BLEU score, RIBES score and AM/FM score.Re-ranking method proves to be best over all our other methods.