Unifying Dynamic Optimizer Search and Network Architecture Search
Binbin Yang, Xiaodan Liang, Junhao Zhong, Jiefeng Peng, Guangrun Wang, Liang Lin · 2021
Network architecture search techniques have recently revolutionized massive application problems due to its advanced performance. However, previous works rely on a strong assumption that the optimizers are manually designed and fixed for all architectures, which deviates from the intuition that each architecture should be empowered with its optimal optimization strategy. In this paper, we propose to unify Dynamic Optimizer Search (DOS) and Network Architecture Search (NAS), aiming to jointly search for the optimization scheduling and network architecture, named as DOS-NAS. The scheduling process of DOS is formulated as a Markov Decision Process, optimized via the REINFORCE algorithm. Conjunct with NAS, our DOS-NAS learns a discrete temporal scheduling to dynamically configure optimizers from a predefined set, leading to personalized optimizer scheduling for the searched network. Extensive experiments on CIFAR-10, CIFAR-100 and Tiny-ImageNet demonstrate the validity of our proposed DOS-NAS in performance improvement and speeding up the convergence significantly. DOS-NAS provides a new paradigm for AutoML systems through the joint search over optimization strategy and network architecture.