Efficient Multi-Fidelity Neural Architecture Search with Zero-Cost Proxy-Guided Local Search
Quan Minh Phan, Ngoc Hoang Luong · Proceedings of the Genetic and Evolutionary Computation Conference · 2024
Using zero-cost (ZC) metrics as proxies for network performance in Neural Architecture Search (NAS) allows search algorithms to thoroughly explore the architecture space due to their low computing costs. Nevertheless, recent studies indicate that relying exclusively on ZC proxies appears to be less effective than using traditional training-based metrics, such as validation accuracy, in seeking high-performance networks. In this study, we investigate the effectiveness of ZC proxies by taking a deeper look into fitness landscapes of ZC proxy-based local searches by utilizing Local Optima Networks (LONs). Our findings exhibit that ZC proxies having high correlation with network performance do not guarantee finding top-performing architectures, and ZC proxies with low correlations could still be better in certain situations. Our results further consolidate the suggestion of favoring training-based metrics over ZC proxies as the search objective. Although we could figure out architectures having the optimal ZC proxy scores, their true performance is often poor. We then propose the Multi-Fidelity Neural Architecture Search (MF-NAS) framework that makes use of the efficiency of ZC proxies and the efficacy of training-based metrics. Experimental results on a wide range of NAS benchmarks demonstrate the superiority of our MF-NAS to state-of-the-art methods under a strict budget.