High Performance Computing: From Deep Learning to Data Engineering
Geoffrey Fox · 2020
We describe how High-Performance Computing (HPC) can be used to enhance Big Data and Machine Learning (ML) systems (HPC for ML) but also how machine learning can be used to enhance system execution (ML for HPC). We review the different aspects of data engineering needed to process large scale data and how it is implemented in the Twister2 system combined with PyTorch and TensorFlow. We discuss the different forms of parallelism seen in deep learning with a focus on pipelined parallelism over layers.