Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback
Khanh Duy Tung Nguyen, Hal Daumé, Jordan Lee Boyd-Graber · 2017
Machine translation is a natural candidate problem for reinforcement learning from human feedback: users provide quick, dirty ratings on candidate translations to guide a system to improve.Yet, current neural machine translation training focuses on expensive human-generated reference translations.We describe a reinforcement learning algorithm that improves neural machine translation systems from simulated human feedback.Our algorithm combines the advantage actor-critic algorithm (Mnih et al., 2016) with the attention-based neural encoderdecoder architecture (Luong et al., 2015).This algorithm (a) is well-designed for problems with a large action space and delayed rewards, (b) effectively optimizes traditional corpus-level machine translation metrics, and (c) is robust to skewed, high-variance, granular feedback modeled after actual human behaviors.