Benchmarking Gradient Estimation Mechanisms in Evolution Strategies for Solving Black-Box Optimization Functions and Reinforcement Learning Problems
Thai Bao Tran, Ngoc Hoang Luong · 2022
In this paper, we investigate the gradient estimation mechanisms of three evolution strategies (ES) algorithms: the vanilla ES (VES), the Guided Evolutionary Strategies (GES), and the Self-Guided Evolution Strategies (SGES). The vanilla ES generates search directions (i.e., its population individuals) following an isotropic Gaussian distribution from the full parameter space, yielding unbiased estimations of the true gradient vectors but suffering from sample inefficiency in high-dimensional problems. GES and SGES aim to construct low-dimensional guiding subspaces, that potentially contain the true gradients, from which search directions for computing gradient estimates can be generated in a more efficient manner. We perform experiments with a variety of high-dimensional optimization problems, including multi-modal black-box functions and noisy reinforcement learning locomotion tasks. Experimental results help pinpoint the essential components of these algorithms and the important issues that need to be considered for their successful applications in solving challenging optimization problems. Source code is available at: https://github.com/ELO-Lab/BenchGEM-ES.