Pareto Local Search is Competitive with Evolutionary Algorithms for Multi-Objective Neural Architecture Search
Quan Minh Phan, Ngoc Hoang Luong · Proceedings of the Genetic and Evolutionary Computation Conference · 2023
Neural architecture search (NAS) involves automatically searching for promising deep neural network structures in certain architecture spaces. Depending on the number of criteria being concerned, NAS can be formulated as single-objective optimization problems (SONAS) or multi-objective optimization problems (MONAS). Evolutionary algorithms (EAs) are common approaches for NAS due to their effectiveness in solving challenging combinatorial problems. Recent studies, however, have analyzed SONAS landscapes and indicated that while NAS problems are multi-modal but local search algorithms with simple perturbation operators can escape local optima to reach global optima without much difficulty. Such investigations for MONAS remain under-explored. In this paper, we employ local optimal networks (LONs) for visually structuring the MONAS landscape with a simple local search procedure. Via detailed analyses, we then design LOMONAS, a dedicated Pareto local search algorithm for MONAS. The experimental results on four NAS benchmarks (MacroNAS, NAS-Bench-101, NAS-Bench-201, and NAS-Bench-ASR) exhibit the superior performance of LOMONAS compared to two widely-used multi-objective EAs (MOEAs), NSGA-II and MOEA/D. The findings indicate that Pareto local search algorithms are competitive with MOEAs in solving MONAS problems.