Efficient Repetition Coding for Deep Learning Towards Implementation Using Emerging Non-Volatile Memory with Write-Errors
Ninnart Fuengfusin, Hakaru Tamukoh, Yuichiro Tanaka, Osamu Nomura, Takashi Morie · 2023
Emerging non-volatile memory devices, such as resistive random access memory (ReRAM) and voltage-controlled magnetoresistive random access memory (VC-MRAM), promise low energy consumption for artificial intelligence applications. However, when implementing deep neural networks (DNNs) using such memory devices, write-error may cause millions of bit-flipping to DNN. This easily degrades the DNN performance. To address this problem, we propose a novel repetition coding for deep-learning (RC-DL), which is a repetition coding designed to protect IEEE 32-bit floating-point (FP32) DNN models. Compared to conventional repetition coding, the proposed RC-DL exploits FP32 non-uniform magnitude encoding by increasing the repeat rates to protect sensitive bit positions and reduce the repeat rates to insensitive bit positions. Hence, RC-DL uses a number of bits equivalent to a 3-bit repetition code while delivering the performance close to 11-bit repetition code. We perform extensive Monte Carlo simulations to simulate the write-error property with ImageNet 2012 pretrained models. The DNN models with RC-DL are shown to be operable in the extremely imperfect environment while delivering with only minor reductions in DNN performance.