A closer look at batch size in mini-batch training of deep auto-encoders
Heng Wang, Kaijun Ren, Junqiang Song · 2017
In deep learning community, gradient based methods are typically employed to train the proposed models. These methods generally operate in a mini-batch training manner wherein a small fraction of the training data is invoked to compute an approximative gradient. It is reported that models trained with large batch are prone to generalize worse than those trained with small batch. Several inspiring works are conducted to figure out the underlying reason of this phenomenon, but almost all of them focus on classification tasks. In this paper, we investigate the influence of batch size on regression task. More specifically, we tested the generalizability of deep auto-encoder trained with varying batch size and checked some well-known measures relating to model generalization. Our experimental results lead to three conclusions. First, there exist no obvious generalization gap in regression model such as auto-encoders. Second, with a same train loss as target, small batch generally lead to solutions closer to the starting point than large batch. Third, spectral norm of weight matrices is closely related to generalizability of the model, but different layers contribute variously to the generalization performance.