A Scalable Multi-TeraOPS Core for AI Training and Inference
Sunil Kumar Shukla, Bruce Fleischer, Matthew M. Ziegler, Joel A. Silberman, Jinwook Oh, Vijayalakshmi Srinivasan, Jungwook Choi, Silvia Melitta Mueller, Ankur Agrawal, Tina Babinsky, Nianzheng Cao, Chia‐Yu Chen, Pierce Chuang, Thomas Fox, George D. Gristede, Michael Guillorn, Howard Haynie, Michael J. Klaiber, Dongsoo Lee, Shih-Hsien Lo · IEEE Solid-State Circuits Letters · 2018
This letter presents a multi-TOPS AI accelerator core for deep learning training and inference. With a programmable architecture and custom ISA, this engine achieves >90% sustained utilization across the range of neural network topologies by employing a dataflow architecture to provide high throughput and an on-chip scratchpad hierarchy to meet the bandwidth demands of the compute units. A custom 16b floating point (fp16) representation with 1 sign bit, 6 exponent bits, and 9 mantissa bits has also been developed for high model accuracy in training and inference as well as 1b/2b (binary/ternary) integer for aggressive inference performance. At 1.5 GHz, the AI core prototype achieves 1.5 TFLOPS fp16, 12 TOPS ternary, or 24 TOPS binary peak performance in 14-nm CMOS.