Measuring Resources and Workload Skew In Micro-service MPP Analytic Query Engine
Nikunj Parek, Swathi Kurunji, Alan Beck · 2018
Big Data analytics is common in many business domains and its uses range from generating simple reports to executing complex analytical workloads. Massively Parallel Processing (MPP) databases address the challenges imposed by big data and containerization technologies makes it easy for businesses to get them on-demand as platform-as-a-service. The use of container as base technology for large-scale distributed systems opens many challenges in the area of resource management at run-time such as, auto-scaling, optimal deployment and monitoring. Monitoring is at the heart of many cloud resource management solutions such as, measuring and optimizing performance, budget planning and management. Measuring Workload Skew and resource utilization of MPP databases and analytic engines is the focus of our work. This paper explores the tools available to measure the performance of MPP Docker and Kubernetes environments from the perspective of a Database Administrator using such systems in a virtualized environment. Proposed approach provides a detailed characterization of CPU, memory and network IO, while complex long duration analytic SQL queries are loading compute resources in containerized systems.