Application-aware networking for big data computing framework
Peng Yang · 2014
As the world enters the era of big data, MapReduce-like data-parallel computing frameworks are widely adopted in clouds and data centers. My objective is to build a more efficient underlying networks and improve the performance of such computing system. Specically, I first present my effort towards comprehensive traffic forecasting for big data applications using external, light-weighted file system monitoring. The idea is motivated by the key observations that rich traffic demand information already exists in the log and meta-data files of many big data applications, and that such information can be readily extracted through run-time file system monitoring. As an initial step, we use Hadoop1 as a concrete example to explore our methodology and develop a system called HadoopWatch to predict traffic demand of Hadoop applications. Our experiments over a series of MapReduce applications demonstrate that HadoopWatch can forecast the traffic demand with almost 100% accuracy and time advance. Meanwhile, it makes no modication of the Hadoop framework, and introduces little overhead to the application performance. Second, I also make my attempt to orchestrate the network with the traffic demand information forecasted by HadoopWatch. After studying the traffic pattern widely existed in these computing frameworks, I realize that many flows are grouped to achieve a common barrier. To diminish the average group communication time, I proposed a task-aware flow scheduling and routing scheme. Flow scheduling is based on Shortest Remaining First scheduling paradigm, while task-aware flow routing can guarantee the grouped flows are evenly distributed across the multiple paths. I design and implement a prototype system called ShuffleBoost, which can cooperate with HadoopWatch and improve network efficiency in Hadoop clusters. In the best case I measured, the average group communication time decreased by 13.5%, and the average job completion time drops by 7.8%. 1One of the most popular and widely-used open-source software framework in cloud computing.