Bandit Structured Prediction for Neural Sequence-to-Sequence Learning
Julia Kreutzer, Artem Sokolov, Stefan Riezler · 2017
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback.This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures.We advance this framework by lifting linear bandit learning to neural sequence-to-sequence learning problems using attention-based recurrent neural networks.Furthermore, we show how to incorporate control variates into our learning algorithms for variance reduction and improved generalization.We present an evaluation on a neural machine translation task that shows improvements of up to 5.89 BLEU points for domain adaptation from simulated bandit feedback.