SpreadOut: A Kernel Weight Initializer for Convolutional Neural Networks
Matheus I. Hertzog, Ulisses Brisolara Corrêa, Ricardo Araújo · 2019
Convolutional Neural Networks are based on the hierarchical extraction of features over many layers, by using convolutional filters, or kernels. Each kernel is coded by a set of weights, which represents a feature to be extracted from the input. On a given layer, it is undesirable to have redundant kernels and the training algorithm must learn to differentiate them. SpreadOut is a kernel weight initializer that differentiates kernels before training, so as to improve convergence rates. It does so by maximizing a distance metric calculated over all pairs of kernels on the same layer, after weights are initialized by a traditional technique. We show that SpreadOut improves convergence rates on several benchmark datasets when compared to traditional approaches.