Mapping Instructions and Visual Observations to Actions with Reinforcement Learning
Dipendra Kumar Misra, John C. Langford, Yoav Artzi · 2017
We propose to directly map raw visual observations and text input to actions for instruction execution.While existing approaches assume access to structured environment representations or use a pipeline of separately trained models, we learn a single model to jointly reason about linguistic and visual input.We use reinforcement learning in a contextual bandit setting to train a neural network agent.To guide the agent's exploration, we use reward shaping with different forms of supervision.Our approach does not require intermediate representations, planning procedures, or training different models.We evaluate in a simulated environment, and show significant improvements over supervised learning and common reinforcement learning variants.