Persistent Contextual Neural Networks for learning symbolic data sequences.
Yann Ollivier · arXiv (Cornell University) · 2013
We introduce persistent contextual neural networks (PCNNs) as a probabilistic model for learning symbolic data sequences, aimed at discovering complex algorithmic dependencies in the sequence. PCNNs are similar to recurrent neural networks but feature an architecture inspired by finite automata and a modified time evolution to better model memory effects. An effective training procedure using a gradient ascent in a metric inspired by Riemannian geometry is developed: this produces an algorithm independent from design choices such as the encoding of parameters and unit activities. This metric gradient ascent is designed to have an algorithmic cost close to backpropagation through time for sparsely connected networks. PCNNs are demonstrated to effectively capture a variety of complex algorithmic constraints on hard synthetic problems: basic block nesting as in context-free grammars (an important feature of natural languages, but difficult to learn), intersections of multiple independent Markovtype relations, or long-distance relationships such as the distant-XOR problem. On this problem, PCNNs perform better than more complex state-of-the-art algorithms. Thanks to the metric update, fewer gradient steps and training samples are needed: for instance, a generating model for sequences of the form