Fast dropout training
Sida Wang, Christopher D. Manning · 2013
Preventing feature co-adaptation by encour-aging independent contributions from differ-ent features often improves classification and regression performance. Dropout training (Hinton et al., 2012) does this by randomly dropping out (zeroing) hidden units and in-put features during training of neural net-works. However, repeatedly sampling a ran-dom subset of input features makes training much slower. Based on an examination of the implied objective function of dropout train-ing, we show how to do fast dropout training by sampling from or integrating a Gaussian approximation, instead of doing Monte Carlo optimization of this objective. This approx-imation, justified by the central limit theo-rem and empirical evidence, gives an order of magnitude speedup and more stability. We show how to do fast dropout training for clas-sification, regression, and multilayer neural networks. Beyond dropout, our technique is extended to integrate out other types of noise and small image transformations. 1.