Stage-wise Training: An Improved Feature Learning Strategy for Deep Models
Elnaz Barshan, Paul Fieguth · Neural Information Processing Systems · 2015
Deep neural networks currently stand at the state of the art for many machine learning applications, yet there still remain limitations in the training of such networks because of their very high parameter dimensionality. In this paper we show that network training performance can be improved using a stage-wise learning strategy, in which the learning process is broken down into a number of related sub-tasks that are completed stage-bystage. The idea is to inject the information to the network gradually so that in the early stages of training the \coarse-scale properties of the data are captured while the erscale characteristics are learned in later stages. Moreover, the solution found in each stage serves as a prior to the next stage, which produces a regularization eect and enhances the generalization of the learned representations. We show that decoupling the classier layer from the feature extraction layers of the network is necessary, as it alleviates the diusion of gradient and over-tting problems. Experimental results in the context of image classication support these claims.