HAL: Computer System for Scalable Deep Learning
Volodymyr Kindratenko, Dawei Mu, Zhan You Yan, J. D. Maloney, Sayed Hadi Hashemi, Benjamin Rabe, Ke Xu, Roy H. Campbell, Jian Peng, William Gropp · Practice and Experience in Advanced Research Computing · 2020
We describe the design, deployment and operation of a computer system built to efficiently run deep learning frameworks. The system consists of 16 IBM POWER9 servers with 4 NVIDIA V100 GPUs each, interconnected with Mellanox EDR InfiniBand fabric, and a DDN all-flash storage array. The system is tailored towards efficient execution of the IBM Watson Machine Learning enterprise software stack that combines popular open-source deep learning frameworks. We build a custom management software stack to enable an efficient use of the system by a diverse community of users and provide guides and recipes for running deep learning workloads at scale utilizing all available GPUs. We demonstrate scaling of a PyTorch and TensorFlow based deep neural networks to produce state-of-the-art performance results.