Recurrent Networks and Natural Language: Exploiting Self-organization
Matthew W. Crocker, Igor Farkaš · eScholarship (California Digital Library) · 2006
Prediction is believed to be an important cognitive component in natural language processing.Within connectionist approaches, Elman's simple recurrent network has been used for this task with considerable success, especially on small scale problems.However, it has been appreciated for some time that supervised gradientbased learning models have difficulties with scaling up, because their learning becomes very time-consuming for larger data sets.In this paper, we explore an alternative neural network architecture that exploits selforganization.The prediction task is effectively split into separate stages of self-organized context representation and subsequent association with the next-word target distribution.We compare various prediction models and show, in the task of learning a language generated by stochastic context-free grammar, that self-organization can lead to higher accuracy, faster training, greater robustness and more transparent internal representations, when compared to Elman's network.