Effect of the Period of the Fourier Series Approximation for Binarized Neural Network
SeonYong Lee, Hee-Youl Kwak, Jong‐Seon No · 2022
The construction of low complexity models for the neural networks is an important issue in practical, real-world scenarios. One of the most famous construction methods for a simple neural network model is to represent weights and activations by the 1-bit quantization, called binarized neural networks (BNNs). However, it is still under research on how to represent the gradient in the backpropagation of BNNs because the activation function is the sign function whose gradients are zero almost everywhere. One way to address this problem is to approximate the gradient of the sign function by the Fourier series representation. In this paper, we analyze the effect of the period and the number of terms of the Fourier series representation on the network accuracy. Since the period has a direct relationship with the degree of the approximation for the sign function and the oscillation behavior of the gradient function, the choice of the period significantly affects the accuracy of the BNN model. The experiments on the CIFAR-10 dataset demonstrate that a proper choice of the period can outperform the conventional BNNs with straight through estimator.