Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks
Brenden M. Lake, Marco Baroni · arXiv (Cornell University) · 2017
Humans can understand and produce new utterances effortlessly, thanks to their systematic compositional skills. Once a person learns the meaning of a new verb dax, he or she can immediately understand the meaning of twice or sing and dax. In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can generalize well when the differences between training and test commands are small, so that they can apply mix-and-match strategies to solve the task. However, when generalization requires systematic compositional skills (as in the dax example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, supporting the conjecture that lack of systematicity is an important factor explaining why neural networks need very large training sets.