Natural Language Grounding and Grammar Induction for Robotic Manipulation Commands
Muhannad Alomari, Paul Duckworth, Majd Hawasly, David Hogg, Anthony G. Cohn · 2017
We present a cognitively plausible system capable of acquiring knowledge in language and vision from pairs of short video clips and linguistic descriptions.The aim of this work is to teach a robot manipulator how to execute natural language commands by demonstration.This is achieved by first learning a set of visual 'concepts' that abstract the visual feature spaces into concepts that have human-level meaning.Second, learning the mapping/grounding between words and the extracted visual concepts.Third, inducing grammar rules via a semantic representation known as Robot Control Language (RCL).We evaluate our approach against state-of-the-art supervised and unsupervised grounding and grammar induction systems, and show that a robot can learn to execute never seenbefore commands from pairs of unlabelled linguistic and visual inputs.