Tuning Hadoop map slot value using CPU and IO metrics
K. C. Kamal, Vincent W. Freeh · NCSU Libraries Repository (North Carolina State University Libraries) · 2013
Hadoop is a widely used open source mapreduce framework.Its performance is critical because it increases the usefulness of products and services for a large number of companies who have adopted Hadoop for their business purposes.One of the configuration parameters that influences the resource allocation and thus the performance of a Hadoop application is map slot value (MSV).MSV determines the number of map tasks that run concurrently on a node.For a given architecture, a Hadoop application has an MSV for which its performance is best.Furthermore, there is not a single map slot value that is best for all applications.A Hadoop application's performance suffers when MSV is not the best.Therefore, knowing the best MSV is important for an application.In this work, we find a low-overhead method to predict the best MSV using two new Hadoop counters that measure per-map task CPU utilization and IO throughput.Our experiments on a wide variety of Hadoop applications show that using the best MSV for each application improves the aggregate performance by 5% up to 132% when compared to using a single MSV for all applications.