Dynamic performance tuning of Hadoop
K. C. Kamal, Vincent W. Freeh · NCSU Libraries Repository (North Carolina State University Libraries) · 2014
Hadoop is an open source mapreduce framework used by a large number of organizations.Its performance is important in increasing the usefulness of large scale data processing applications.Map slot value (MSV) is one of the configuration parameters that influences the resource allocation and the performance of a Hadoop application.MSV determines the number of map tasks that can run concurrently on a node.In this work, we develop an approach to dynamically change MSV during the execution of an application.Our approach converges to the best MSV for all types of applications and it does so with low overhead.Without dynamic approach, determining the best MSV of an application requires a very tedious process of measuring the map completion time for all MSV settings.Our approach also adjusts MSV for applications that may not have a single best MSV throughout their execution.Compared to the peformance of an application when it is using the best MSV, performance of our dynamic approach is within 4.6% with cold start and improves by as much as 5% with warm start.