HyperFDA
Léo Souquet, Nadiya Shvai, Arcadi Llanza, Amir Nakib · 2020
This paper introduces Hyper-FDA: the application of the meta-heuristic "Fractal Decomposition Algorithm" (FDA) to the optimization of the hyperparameters of deep neural network architectures. FDA is a metaheuristic that was recently proposed to solve high dimensional continuous optimization problems. This approach is based on a geometric fractal decomposition which divides the search space while looking for the optimal solution. The presented approach uses bi-level optimization to separate the architecture search composed of discrete parameters from hyperparameters optimization with continuous parameters. The architecture search is conducted by a random walker aiming to generate chained-structured architectures. The best ones are selected to be fined tuned using FDA. The benchmark CIFAR-10 is used to test the approach and results are among the state of the art considering the low number of parameters and the chained-structured architecture of the network. The encouraging results are emphasized by the fact that the entire process was conducted using low computational power with only 3 GPUs.