Asymptotic Spectral Representation of Linear Convolutional Layers
Xinping Yi · IEEE Transactions on Signal Processing · 2022
By stacking a number of convolutional layers, convolutional neural networks (CNNs) have made remarkable performance boosts in many artificial intelligence applications. While the convolution operation is well-understood, it is still a mystery why repeated convolutions yield so good expressive power and generalization performance. Noting that the linear convolution operation can be represented as a matrix-vector product with the matrix being of a Toeplitz structure, we propose to inspect the individual convolutional layer through its asymptotic spectral representation - the spectral density matrix - by leveraging Toeplitz matrix theory. Thanks to such spectral representation, we are able to develop a simple singular value approximation method with improved accuracy, and spectral norm upper bounds with reduced computational complexity, compared with the state-of-the-art methods. Both the improved approximation and upper bounds can be employed as regularization techniques to further enhance the generalization performance of CNNs. By extensive experiments on well-deployed CNN models (e.g., ResNets), we also demonstrate that the approximation approach achieves higher accuracy and the upper bounds are effective spectral regularizers for generalization.