Data-Dependent Path Normalization in Neural Networks
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, Nathan Srebro · arXiv (Cornell University) · 2015
We propose a unified framework for neural net normalization, regularization and optimization, which includes Path-SGD and Batch-Normalization and interpolates between them across two different dimensions. Through this framework we investigate issue of invariance of the optimization, data dependence and the connection with natural gradients.