LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications
Jinhan Xin, Kai Hwang, Zhibin Yu · Proceedings of the 2022 International Conference on Management of Data · 2022
Spark SQL has been widely deployed in industry but it is challenging to tune its performance. Recent studies try to employ machine learning (ML) to solve this problem, but suffer from two drawbacks. First, it takes a long time (high overhead) to collect training samples. Second, the optimal configuration for one input data size of the same application might not be optimal for others.