Integrating apache spark into PBS-Based HPC environments
Troy Baer, Paul Peltz, Junqi Yin, Edmon Begoli · 2015
This paper describes an effort at the University of Tennessee's National Institute for Computational Sciences (NICS) to integrate Apache Spark into the widely used TORQUE HPC batch environment. The similarities and differences between the execution of a Spark program and that of an MPI program on a cluster are used to motivate how to implement Spark/TORQUE integration. An implementation of this integration, pbs-spark-submit, is described, including demonstrations of functionality on two HPC clusters and a large shared-memory system.