Accelerator-centric deep learning systems for enhanced scalability, energy-efficiency, and programmability
Minsoo Rhu · 2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC) · 2018
Deep learning (DL) has been successfully deployed in various application domains ranging from computer vision, speech recognition, to natural language processing. As the network models and the datasets used to train these models scale, system architects are faced with new challenges in designing a scalable and energy-efficient high-performance computing (HPC) system for training DL algorithms. One of the key obstacles that DL researchers are facing is the memory capacity bottleneck, where the limited physical memory size of the PCIe-attached DL accelerator (whether it be a discrete GPU or ASIC accelerator like Google's Tensor Processing Unit) constrains the algorithm that can be studied. In this paper, and the associated invited special session talk, we first discuss recent research literature geared towards designing scalable HPC systems for DL. In this context, we then discuss the memory capacity wall problem and introduce the work on virtualized deep neural networks, a memory virtualization solution that systematically reduces the memory consumption of DNN training. We conclude by providing projections on future challenges DNN memory virtualization will encounter upon and suggest accelerator-centric DL system as a promising research direction for the development of a scalable and energy-efficient deep learning system architecture.