Accelerating Big Data Infrastructure and Applications (Ongoing Collaboration)
Kevin Brown, Tianqi Xu, Keita Iwabuchi, Kento Sato, Adam T. Moody, Kathryn Mohror, Nikhil Jain, Abhinav Bhatelé, Martin Schulz, Roger Pearce, Maya B. Gokhale, Satoshi Matsuoka · 2017
High-performance computing (HPC) systems are increasingly being used for data-intensive, or "Big Data", workloads. However, since traditional HPC workloads are compute-intensive, the HPC-Big Data convergence has created many challenges with optimizing data movement and processing on modern supercomputers. Our collaborative work addresses these challenges using a three-pronged approach: (i) measuring and modeling extreme-scale I/O workloads, (ii) designing a low-latency, scalable, on-demand burst-buffer solution, and (iii) optimizing graph algorithms for processing Big Data workloads. We describe the three areas of our collaboration and report on their respective developments.