Training Neural Networks with Low Precision Dynamic Fixed-Point
Sujeong Jo, Hanmin Park, Gunhee Lee, Ki‐Young Choi · 2018
Dynamic fixed-point (DFP) is one of the most successful attempts to reduce bit-widths in training neural networks. It has been reported that DFP can reduce the bit-widths of the training operations to 16 bits, except for parameter update operations; parameter updates for general networks need higher precision and are usually done with 32-bit floating-point operations. In this paper, we propose two methods of using 16-bit DFP for all the training operations including parameter updates; weight clipping and gradual batch size increase. Lastly, we combine the two methods to further explore their potentials. We successfully apply 16-bit DFP operations on the parameter updates of LeNet-5 and VGG-16 networks using CIFAR10 and CIFAR100 datasets.