Sequential learning for extreme learning machine
Nanying Liang · 2006
The approximation capability of feedforward neural networks has been deeply studied, and it has been shown that the neural network with as few as a single hidden layer is a universal approximator.We refer such neural networks as Single hidden Layer Feedforward Neural networks (SLFNs).Depending on the kind of activation function used in the hidden layer, two popular structures of SLFNs can be recognized in the literature: SLFNs with ridge activation function, and SLFNs with radial basis activation function: radial basis function neural networks (RBF-NNs).By recognizing the need for sequential learning in practice, we lay the foundation of the thesis on sequential learning algorithm design for SLFNs.Traditional sequential algorithms for training SLFNs with ridge activation function are referred to as stochastic BP-based algorithms.For training RBF-NNs, they are referred to as RAN-based algorithms.However, the intrinsic drawbacks of these sequential learning algorithms prohibit their applications into complicate and large-scale problems.This motivates us to propose a novel method, which is referred to as Online Sequential Extreme Learning Machine (OS-ELM) in the thesis.OS-ELM is derived from batch Extreme Learning Machine (ELM) on the basis of recursive least-squares (RLS) algorithm.The resulted sequential algorithm trains both SLFNs with ridge function and RBF-NNs in a unified framework.Additionally it can operate in the learning mode of one-by-one as well as chunk-by-chunk.The implementation of OS-ELM is composed of two phases, viz., initialization and update phase.In the initialization phase of the algorithm, the values for the hidden unit parameters of a SLFN are randomly assigned.In the update phase, v