Feature Extraction Mechanism for Each Layer of Deep Echo State Network
Keikou Kanda, Sou Nobukawa · 2022
The echo state network (ESN) is an efficient machine learning model that is the most typical type of reservoir-computing framework. Recently, research has been conducted on deep echo state networks (deepESN). The deepESN consists of an input layer, multiple reservoir layers, and an output layer, which achieve a very high memory capacity (MC). Furthermore, it has been suggested that deepESN can represent various temporal scales using layer hierarchization. However, the exact role of each layer in the feature extraction has not yet been revealed. Therefore, deepESN parameter adjustments must be conducted using empirical measurements or grid searches based on a trial-and-error method. To establish a design framework for deepESN, revealing the deepESN parameters related to feature extraction is crucial. To analyze the dynamics of neural networks, we applied multiscale entropy (MSE) analysis to a physiological neural network model and found that complex topological features and multiple neural module structures produce complex temporal-scale dependencies. Therefore, we hypothesized that the feature extraction function of each layer in the deepESN could be revealed using MSE analysis. To validate this hypothesis, we analyzed the output of each layer using MSE analysis and MC task. As a result, in this study, under small inner layer connections, a high memory capacity was achieved by temporal scale-specific feature extraction in each layer compared to larger inter-layer connections. In conclusion, this study revealed the feature extraction function in deepESN and provided the parameter setting method, especially regarding interlayer connection as a part of the design framework of deepESN.