A Tale about PRO and Monsters
Preslav Nakov, Francisco Guzmán, Stephan Vogel · 2013
While experimenting with tuning on long sentences, we made an unexpected discov-ery: that PRO falls victim to monsters – overly long negative examples with very low BLEU+1 scores, which are unsuitable for learning and can cause testing BLEU to drop by several points absolute. We propose several effective ways to address the problem, using length- and BLEU+1-based cut-offs, outlier filters, stochastic sampling, and random acceptance. The best of these fixes not only slay and pro-tect against monsters, but also yield higher stability for PRO as well as improved test-time BLEU scores. Thus, we recommend them to anybody using PRO, monster-believer or not. 1 Once Upon a Time... For years, the standard way to do statistical ma-chine translation parameter tuning has been to use minimum error-rate training, or MERT (Och, 2003). However, as researchers started using mod-els with thousands of parameters, new scalable op-timization algorithms such as MIRA (Watanabe et