Building predictive models on complex symbolic sequences with a second-order recurrent BCM network with lateral inhibition
Peter Tiňo, Michal Stančík, Ľubica Beňušková · 2000
When trained on symbolic sequences to perform the next-symbol prediction, recurrent neural networks (RNNs) tend to organize their state space so that "close" recurrent activation vectors correspond to histories of symbols yielding similar next-symbol distributions. In this paper we investigate an unsupervised alternative to the state space organization. In particular, we use a recurrent version of the Bienenstock-Cooper-Munro (BCM) network with lateral inhibition to map histories of symbols into activations of the recurrent layer. Recurrent BCM networks perform a kind of time-conditional projection pursuit. We compare the finite-context models built on top of BCM recurrent activations with those constructed on top of RNN recurrent activation vectors. As a test bed we use two complex symbolic sequences with rather deep memory structures. It is shown that the BCM-based model has a comparable or better performance than its RNN-based counterpart. This can be explained by the familiar information latching problem in recurrent networks when longer time spans are to be latched.