Generalization and Systematicity in Echo State Networks
Stefan Leo Frank, Micheal Čerňanský, Love, B.C., McRae, K., Sloutsky, V.M. · eScholarship (California Digital Library) · 2008
Echo state networks (ESNs) are recurrent neural networks that can be trained efficiently because the weights of recurrent con-nections remain fixed at random values. Investigations of these networks ’ ability to generalize in sentence-processing tasks have resulted in mixed outcomes. Here, we argue that ESNs do generalize but that they are not systematic, which we define as the ability to generally outperform Markov models on test sentences that violate the training sentences ’ grammar. More-over, we show that systematicity in ESNs can easily be ob-tained by switching from arbitrary to informative representa-tions of words, suggesting that the information provided by such representations facilitates connectionist systematicity.