Empirical Study on the Impact of Few-Cost Proxies
Kevin Kollek, Marco Braun, Jan-Hendrik Meusener, Jan-Christoph Krabbe, Anton Kummert · 2024
Selecting an optimal neural network architecture tailored to a specific dataset is a time-consuming task due to numerous design possibilities. Neural Architecture Search (NAS) provides strategies to identify well performing networks in a limited timeframe. Zero-cost proxies offer a training-free approach to find potential architectures within a predefined search space. However, relying solely on these proxies often leads to unreliable results across diverse search spaces and datasets. In this paper, we present an empirical study of extended zero-cost proxies, termed few-cost proxies, obtained by training for a restricted number of epochs. Our analysis demonstrates that these few-cost proxies significantly enhance the ranking performance. Furthermore, novel few-cost proxies introduced in this study outperform previous methods significantly, achieving a Spearman correlation of 0.89 compared to the second-highest score of 0.847 on TSS-Cifar10, showcasing their effectiveness in the context of NAS.