HaT: Holding Oscillation for Efficient INT8 Quantized CNN Training
Chanyung Kim, Dongyeob Shin, Minkyu Lee, Sung‐Joon Jang, Sang-Seol Lee · 2025
Reducing model capacity through low-bit quantization while maintaining performance is a challenging task in deep learning. Quantization during training is particularly difficult compared to quantization in inference models due to issues such as small gradients and quantization errors resulting from weight rounding. A significant issue is weight oscillations, where weights fluctuate between adjacent quantization levels. This introduces noise, leading to decreased training stability and performance degradation. Various methods have been proposed to address this problem. However, most focus on stabilizing the later stages of training or introduce additional computational overhead due to complex algorithms. This study approaches the oscillation phenomenon from a more detailed perspective, focusing on layer-wise, channel-level oscillations throughout the entire training process and their impact on training instability. The proposed Holding and Training (HaT) method continuously monitors oscillations from the early stages of training, temporarily holding channels with high variance, and focusing on the training of the remaining channels. This strategy enhances training stability while simultaneously reducing computational cost. Applying the proposed HaT method to ResNet-20 resulted in up to a 3% improvement in performance and up to a 2 % increase in training speed across various datasets.