What do RNN Language Models Learn about Filler–Gap Dependencies?
Ethan Wilcox, Roger Lévy, Takashi Morita, Richard Futrell · 2018
RNN language models have achieved stateof-the-art perplexity results and have proven useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they learn.Here we investigate whether state-ofthe-art RNN language models represent longdistance filler-gap dependencies and constraints on them.Examining RNN behavior on experimentally controlled sentences designed to expose filler-gap dependencies, we show that RNNs can represent the relationship in multiple syntactic positions and over large spans of text.Furthermore, we show that RNNs learn a subset of the known restrictions on filler-gap dependencies, known as island constraints: RNNs show evidence for wh-islands, adjunct islands, and complex NP islands.These studies demonstrates that stateof-the-art RNN models are able to learn and generalize about empty syntactic positions.