Variance based Averaging and Standard Normal Distribution for Improvement of SBT Quantization in Neural Network Coding
Minseok Lee, Seongbae Rhee, Kyuheon Kim · 2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS) · 2022
Deep Neural Network (DNN) models are undergoing active development, and the efforts to achieve higher task performance have caused models to become more complex and larger in size. This limitation causes difficulties of being used in low-power edge devices. To solve this issue, MPEG (Moving Picture Experts Group) of ISO/IEC has developed NNC (Neural Network Coding), a standard that can reduce model complexity and size, thus making facilitation of DNN models in low-power edge devices possible. This standard is designed as a toolbox of compression methods, including Stochastic Binary Ternary (SBT) quantization. SBT quantization is an approximation method that achieves extremely high coding gains by approximating all non-zero values within a weight tensor to the tensor’s mean. However, this technique can cause significant performance loss due to its inaccurate estimate of values. We therefore propose Variance based Optimal Averaging (VOA) and Standard Normal Distribution based optimization (SNDO), which are two different skills to enhance the performance of SBT quantization by searching for more accurate estimates of parameters. In this paper, we explain the mechanism of these two methods, and confirm their performance through experimental results.