An Experimental Comparison Between Genetic Algorithm and Particle Swarm Optimization in Spark Performance Tuning
Yuzhao Wang, Qixiao Liu, Junqing Yu, Zhibin Yu · 2017
The most popular in-memory computing framework --- Spark --- has a number of performance-critical configuration parameters. Manually tuning these parameters for optimized performance is not practical because the parameter tuning space is huge. Searching algorithms such as genetic algorithm can be used to automatically search the optimal configurations. However, there are several such algorithms and it is unclear which one is better in the case of Spark configuration parameter tuning.