The Importance of Generation Order in Language Modeling
Nic Ford, Daniel Duckworth, Mohammad Norouzi, George E. Dahl · 2018
Neural language models are a critical component of state-of-the-art systems for machine translation, summarization, audio transcription, and other tasks.These language models are almost universally autoregressive in nature, generating sentences one token at a time from left to right.This paper studies the influence of token generation order on model quality via a novel two-pass language model that produces partially-filled sentence "templates" and then fills in missing tokens.We compare various strategies for structuring these two passes and observe a surprisingly large variation in model quality.We find the most effective strategy generates function words in the first pass followed by content words in the second.We believe these experimental results justify a more extensive investigation of generation order for neural language models.