Nimble: Mobile Interface for a Visual Question Answering Augmented by Gestures
Yevhen Romaniak, Anastasiia Smielova, Yevhenii Yakishyn, Valerii Dziubliuk, Mykhailo Zlotnyk, Oleksandr Viatchaninov · 2020
Virtual Assistants are becoming increasingly popular. However voice-only systems providing limited functionality and minimal variability are often unusable because the exposed behavior is either fully deterministic or essentially ambiguous for supporting human-like dialogues. This paper introduces a Nimble, solution that allows users to ask short questions and to get answers about objects selected from a scene by natural pointing gestures. With a modified Visual Question Answering model we have shown how the integration of gestures to the attention mechanism can reduce questions? ambiguity while sustaining the same accuracy level of the system. We performed this by modifying the model's attention scores using gestures fused with linguistic information.