Efficient Search for Efficient Architecture
Liewen Liao, Yaoming Wang, Hao Li, Wenrui Dai, Chenglin Li, Junni Zou, Hongkai Xiong · 2022
Differentiable architecture search (DARTS) has achieved success in searching powerful network architectures but suffers from unstable search, high search cost from repetitive attempts and unawareness of computational cost. In this paper, we propose a novel approach, namely Efficient and Stable Differentiable Architecture Search (ES-DARTS), that leverages decoupled search strategy and variational proxy pruning to achieve efficient search for efficient neural networks. Specifically, the decoupled search strategy stabilizes the search via bridging the gap between search and evaluation, and achieves acceleration with decoupled optimization, while the variational proxy pruning introduces structural pruning into DARTS to accommodate varying constraints on computational cost. ES-DARTS can be a plug-and-play module for DARTS-based approaches to achieve improved performance with reduced search cost and FLOPS. Experimental results demonstrate that ES-DARTS reduces top-1 error rate to 2.71% with $10 \times$ acceleration of DARTS on CIFAR10. ES-DARTS finds an architecture of 2.80% top-1 error rate with only 2.27 MB parameters on CIFAR-10 and of 29.4% top-1 error rate with only 322M FLOPS when transferred to ImageNet.