Firebird: Network-Aware Task Scheduling for Spark Using SDNs
Xin He, Prashant J. Shenoy · 2016
Recently Spark has become a popular cluster computing platform because of its fast in-memory computing which allows users to cache data in servers' memory and query it repeatedly. However, since network I/O is much slower than local I/O, the network can be a bottleneck for data intensive jobs. When data locality is not well balanced or network capacity is limited, data contention can occur, which will significantly slow down task execution. The current delay scheduling method in Spark cannot perfectly solve this problem because it is agnostic to the network status in a cluster. In this paper, we propose a network-aware scheduling method in Spark and design Firebird, a derivative of Spark that runs on top of software-defined network (SDN). By using SDNs, the communication barriers between cluster computing platform and underlying networking are removed and tasks can be scheduled based on network conditions in Spark clusters. We demonstrate the effectiveness of the methods through detailed experiments with different types of jobs on our system. Experimental results show significant improvement of data-intensive jobs, and our system can achieve best scheduling in different cases without tuning Spark. Firebird can be up to 9 times faster than default Spark in some cases.