Conceptor-based deep neural networks
Guangwu Qian, Lei Zhang, Yan Wang · Scientia Sinica Informationis · 2018
In recent years, deep neural networks, also known as deep learning, have achieved several breakthroughs in different fields that were previously dominated by machine learning. Even when using high-performance computing devices, it takes days or weeks to train a deep neural network. Conceptor, as an extension of echo state networks, can be understood as certain neural filters that characterize dynamical neural activation patterns. In this study, based on some improvements to the original conceptor model, we have conducted several studies from the perspectives of non-iterative methods and transfer learning to address the issues mentioned above, which can be summarized as follows: (1) A conceptor-based classifier for non-temporal data and a non-iterative approach feedforward convolutional conceptor neural network are proposed. This classifier achieves classifying accuracy comparable to that of the state-of-the-art methods while requiring significantly less training time. Through experiments on MNIST variation datasets, we evaluate the classifying quality of the feedforward convolutional conceptor neural network. (2) A classifier called fast conceptor classifier is proposed based on conceptors and it achieves state-of-the-art results with the training time reduced by a factor of 60 on average. Its evaluations with pre-trained rather than fine-tuned neural networks have been investigated on Caltech-101 and Caltech-256 datasets.