A Reconfigurable High-Precision and Energy-Efficient Circuit Design of Sigmoid, Tanh and Softmax Activation Functions
Bochang Wang, Ziang Duan, Zixuan Shen, Yuansheng Zhao, Lu Gao, Chao Wang · 2023
This paper proposes a reconfigurable circuit design for Sigmoid, Tanh and Softmax nonlinear functions using Coordinate Rotation Digital Computer (CORDIC) algorithm for deep neural network hardware accelerators. By exploiting arithmetic operation similarity, the proposed reconfigurable design reuses CORDIC units to implement the exponent and division operations in the nonlinear functions, and also utilizes the successive approximation property of CORDIC to achieves adjustable accuracy. In addition, Fast-Convergence CORDIC algorithm is employed to reduce the redundant iterations so as to effectively achieve high accuracy and low power consumption. FPGA implementation shows that it can achieve an adjustable precision range of$\mathbf{1}\times \mathbf{10^{-1}}-\mathbf{3}\times \mathbf{10^{-3}}$with 948 LUTs and 189 FFs. At 100 MHz, it outperforms the conventional standalone designs, i.e., 42.6% and 26.5% increase in precision, 40.5%, 41.9% and 48.9% reduction in latency, and 21.5%, 18.5% and 9.6% increase in energy efficiency, respectively. The preliminary evaluation results in a case study shows that the accuracy loss of LSTM neural network using the proposed design is only 0.98% on Google Speech Command v2 keyword recognition task.