Progressive Variable Precision DNN With Bitwise Ternary Accumulation
Junnosuke Suzuki, Mari Yasunaga, Kazushi Kawamura, Thiem Van Chu, Masato Motomura · 2024
Progressive variable precision networks are capable of adapting to changing computational needs over time using a single weight set. However, previous works have two problems: 1) the absence of zero representation, which limits potential performance gains, and 2) significant accuracy degradation at low bitwidths. To address these issues, this work proposes bitwise ternary (BWT) quantization that progressively accumulates ternary weights based on Booth encoding, achieving a stepwise representation extension from ternary to N-bit. Moreover, the low-bit-first training effectively minimizes the significant accuracy degradation and improves accuracy-computation tradeoff in lower bitwidths. The evaluation results show that the ternary model achieves 76.2 % accuracy with little loss at 8-bit for ResNet18 on CIFAR-100. Similarly, on ImageNet, the model achieves an accuracy of 61.4 % using a ternary weight with less than 1 % degradation at the 8-bit model. Remarkably, these results are obtained with a single weight set.