A Framework for Characterizing Very Large Cloud Workload Traces with Unsupervised Learning
Basem Suleiman, Mohammed Mustafa Fulwala, Albert Y. Zomaya · 2023
The rapid expansion of cloud infrastructure services as dominating computing paradigm across diverse applications has introduced new complexities in the allocation of on-demand cloud resources. Efficiently provisioning computing resources for various job types with distinct requirements poses a fundamental challenge for cloud infrastructure providers, who are responsible for managing data centers. In this paper, we propose a new clustering framework for large-scale workload traces from the Google cloud platform. Our approach intelligently identifies groups of similar jobs by considering key factors such as the centroid of clusters (representing CPU and memory usage deviation), fluctuations in CPU and memory usage, and the time distribution updates following each clustering step. We evaluate our proposed clustering approach by conducting empirical analysis on a substantial dataset consisting of 2.4 TB workload traces from eight different Borg cells within Google's cluster infrastructure. Through this evaluation, we derived seven key observations that shed light on the characteristics of the workload traces. These observations encompass diverse job types, resource utilization patterns, the efficiency of current resource allocation strategies, and the duration of resource usage for different job types and priorities. The insights gained from these observations can provide empirical evidence and valuable guidance for both cloud providers and researchers in designing optimal resource allocation and management strategies for large-scale cloud data centers.