I/O Throttling and Coordination for MapReduce
Siyuan Ma, Xian‐He Sun, Ioan Raicu · 2012
Abstract — As a leading framework for data intensive computing, MapReduce has gained enormous popularity in large-scale data analysis. With the increasing adoption of multi/many core platform, more and more MapReduce tasks are now running on the same node and sharing the same storage resources. The concurrency of tasks raises the issue of I/O stream congestion. We have observed significant throughput drops and task delays caused by I/O stream congestion in the MapReduce framework. In this paper, we propose two techniques to address the I/O stream congestion in MapReduce tasks. First, I/O stream throttling is presented to limit the number of concurrent I/O streams, and avoid throughput drops. Furthermore, to alleviate the I/O contention among multiple MapReduce jobs, I/O coordination orders the I/O streams in accordance to job priority. By exclusively granting I/O resources to streams with higher priorities, the coordination effectively shortens the average job completion time. Experimental results from Hadoop confirm that the proposed techniques improve the average job completion time by up to 33.74%. In addition, the proposed techniques greatly accelerate the execution of high priority jobs; thereby, showing it is capable of fostering QoS in the MapReduce framework. Keywords- I/O stream; MapReduce; I/O scheduling; throttling; coordination I.