A preliminary study of incorporating GPUs in the Hadoop framework
Amin Abbasi, Farshad Khunjush, Reza Azimi · 2012
Fine-grained parallel processors can be employed as accelerators in MapReduce clusters to improve the completion time of MapReduce jobs or to substantially reduce the size of the clusters required to achieve a desired parallel speedup. However, significant architectural differences between conventional CPUs and accelerators pose new challenges for effective scheduling of MapReduce tasks on individual cluster nodes. In this paper, we present a Hadoop-based framework that allows employing both CPUs and GPUs for MapReduce-type applications. We base a novel framework, called Surena, on existing work that allows writing MapReduce applications for GPUs and they incorporate it in the overall Hadoop framework. In particular, we show that by using simple scheduling optimizations, Surena can fully utilize GPUs during the map phase of MapReduce jobs which is often the dominant component in the total execution time of MapReduce applications. Our performance results shows speedups of up to 21x for our framework compare to Hadoop.