Work-in-Progress: NAPMAE: Generalized Data-Efficient Neural Architecture Predictor with Masked Autoencoder
Qiaochu Liang, Lei Gong, Chao Wang, Xuehai Zhou, Xi Li · 2023
Predictor-based Neural Architecture Search (NAS) offers a promising solution for enhancing the efficiency of traditional NAS methods. However, it is non-trivial to train the predictor with limited architecture evaluations for efficient NAS. In this paper, we propose a self-supervised transformer-based model that leverages unlabeled data to learn meaningful representations of neural architectures. After pre-training with a masked autoencoder, the predictor can adapt to various performance metrics estimation with limited supervised data. Experimental results demonstrate that our predictor requires less labeled data and achieves superior performance compared to existing predictors.