Privacy-Preserving Sharing of Data Analytics Runtime Metrics for Performance Modeling
Jonathan Will, Dominik Scheinert, Seraphin Zunzer, Jan Bode, Cedric Kring, Lauritz Thamsen · 2024
Performance modeling for large-scale data analytics workloads can improve the efficiency of cluster resource allocations and job scheduling. However, the performance of these workloads is influenced by numerous factors, such as job inputs and the assigned cluster resources. As a result, performance models require significant amounts of training data. This data can be obtained by exchanging runtime metrics between collaborating organizations. Yet, not all organizations may be inclined to publicly disclose such metadata.