Training an adaptive dialogue policy for interactive learning of visually grounded word meanings
Yanchao Yu, Arash Eshghi, Oliver Lemon · 2016
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor.The system integrates an incremental, semantic parsing/generation framework -Dynamic Syntax and Type Theory with Records (DS-TTR) -with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces.We use this system in interaction with a simulated human tutor to study the effects of different dialogue policies and capabilities on accuracy of learned meanings, learning rates, and efforts/costs to the tutor.We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues;(2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical and incrementally constructed dialogue turns.Ultimately, we train an adaptive dialogue policy which optimises the trade-off between classifier accuracy and tutoring costs.