The Restricted Isometry Property for Echo State Networks with applications to sequence memory capacity
Han Lun Yap, Adam S. Charles, Christopher J. Rozell · 2012
The ability of networked systems (including artificial or biological neuronal networks) to perform complex data processing tasks relies in part on their ability to encode signals from the recent past in the current network state. Here we use Compressed Sensing tools to study the ability of a particular network architecture (Echo State Networks) to stably store long input sequences. In particular, we show that such networks satisfy the Restricted Isometry Property when the input sequences are compressible in certain bases and when the number of nodes scale linearly with the sparsity of the input sequence and logarithmically with its dimension. Thus, the memory capacity of these networks depends on the input sequence statistics, and can (sometimes greatly) exceed the number of nodes in the network. Furthermore, input sequences can be robustly recovered from the instantaneous network state using a tractable optimization program (also implementable in a network architecture).