Unsupervised Learning in Recurrent Neural Networks
Magdalena Klapper-Rybicka, Nicol Norbert Schraudolph, Jürgen Schmidhuber · 2000
. While much work has been done on unsupervised learning in feedforward neural network architectures, its potential with (theoretically more powerful) recurrent networks and time-varying inputs has rarely been explored. Here we train Long Short-Term Memory (LSTM) recurrent networks to maximize two information-theoretic objectives for unsupervised learning: Binary Information Gain Optimization (BINGO) and Nonparametric Entropy Optimization (NEO). LSTM learns to discriminate di erent types of temporal sequences and group them according to a variety of features. 1 Introduction Unsupervised detection of input regularities is a major topic of research on feedforward neural networks (FFNs), e.g., [1-33]. Most of these methods derive from information-theoretic objectives, such as maximizing the amount of preserved information about the input data at the network's output. Typical real-world inputs, however, are not static but sequential, full of temporally extended features, statist...