Pruning Algorithm for Neural Networks Based on Pseudo-Entropy of Weights
Huizhong Yang · Jisuanji fangzhen · 2006
A common method for combating over-fitting problem is to apply pruning to reduce the number of unnecessary weights. By introducing the pseudo-entropy of weights as a penalty-term into the normal objective function, the distribution of weights is constrained during training, and in the training process, the sensitivity of weights is served as the criteria of pruning to avoid the randomicity of pruning only by the size of the weights. The small connections that have the smaller sensitivity value will be pruned therefore it is very effective because no retrain is required after pruning. The simulation result shows that it is simple, cheap to be implemented and the generalization of feed-forward neural networks trained by the proposed algorithm is greatly improved.