$\ell _1$ Regularization in Two-Layer Neural Networks
Gen Li, Yuantao Gu, Jie Ding · IEEE Signal Processing Letters · 2021
A crucial problem of neural networks is to select an architecture that strikes appropriate tradeoffs between underfitting and overfitting. This work shows that$\ell _1$regularizations for two-layer neural networks can control the generalization error and sparsify the input dimension. In particular, with an appropriate$\ell _1$regularization on the output layer, the network can produce a tight statistical risk. Moreover, an appropriate$\ell _1$regularization on the input layer leads to a risk bound that does not involve the input data dimension. The results also indicate that training a wide neural network with a suitable regularization provides an alternative bias-variance tradeoff to selecting from a candidate set of neural networks. Our analysis is based on a new integration of dimension-based and norm-based complexity analysis to bound the generalization error.