Dynamical Isometry Based Rigorous Fair Neural Architecture Search
Jianxiang Luo, Junyi Hu, Weiwei Li, Guanghui Cheng · 2026
While weight-sharing has accelerated Neural Architecture Search (NAS), ensuring fair evaluation remains a critical challenge. For instance, fast methods like BNNAS freeze network weights for efficiency, but this approach lacks theoretical grounding, often leading to unreliable module evaluations. To address this, we introduce a novel and principled NAS framework grounded in the principles of dynamical isometry. We provide a theoretical analysis, demonstrating that initializing a weight-sharing Supernet to be isometric stabilizes the information flow, thereby creating a more fair environment for evaluation. Building on this stable foundation, we prove that the learnable parameters of a strategically placed Batch Normalization (BN) layer serve as a robust proxy for the quality of its preceding frozen module by controlling the concentration of the module’s output distribution. Extensive experiments validate our approach: architectures discovered by our method achieve state-of-the-art Top-1 accuracy on ImageNet in the lightweight regime, and our framework exhibits significantly more stable and reliable search dynamics compared to previous indicator-based methods.