Resource Management for Systems Running MapReduce Jobs
Shikharesh Majumdar · 2024
Chapter 6 concerns data-intensive distributed systems and focuses on platforms running MapReduce jobs that are used in big data analytics as well as for other data-intensive applications. This chapter describes techniques for allocation and scheduling for MapReduce jobs. Algorithms for systems on which jobs are to be completed on a best effort basis are described first. Resource management on systems for which jobs are associated with service level agreements (SLAs) that include job completion deadlines is described next. Two such resource management algorithms, a budget-based algorithm and a constraint-programming-based algorithm are discussed. The SLA associated with a job includes user estimates of task execution times that are often subject to error. Two techniques for handling such errors and increasing the robustness of resource management algorithms are described. The chapter includes a thorough discussion of the performance of the various techniques described.