A Spoken Dialogue System for Spatial Question Answering in a Physical Blocks World
Georgiy Platonov, Lenhart K. Schubert, Benjamin Kane, Aaron Gindi · 2020
A physical blocks world, despite its relative simplicity, requires (in fully interactive form) a rich set of functional capabilities, ranging from vision to natural language understanding.In this work we tackle spatial question answering in a holistic way, using a vision system, speech input and output mediated by an animated avatar, a dialogue system that robustly interprets spatial queries, and a constraint solver that derives answers based on 3-D spatial modeling.The contributions of this work include a semantic parser that maps spatial questions into logical forms consistent with a general approach to meaning representation, a dialogue manager based on a schema representation, and a constraint solver for spatial questions that provides answers in agreement with human perception.These and other components are integrated into a multi-modal human-computer interaction pipeline.