Too-Hot-to-Handle: Insights into Temperature and Noise Hyperparameters for Differentiable Neural-Architecture-Searches
Joschua Conrad, Simon Wilhelmstätter, Rohan Asthana, Vasileios Belagiannis, Maurits Ortmanns · 2024
The deployment of neural networks on resource-constrained hardware requires architecture optimization algorithms that respect accuracy and computational cost. One emerging way for performing such an architecture optimization is the differentiable neural-architecture-search (DNAS). DNAS finds the weights of a neural network as well as its topology in a single training. The topology search-space is represented by instantiating different layer-candidates for each layer in parallel, thereby forming a so-called weighted super-network. The instances are connected via a gumbel-softmax function which employs important parameters commonly called temperature τ and noise-scale β. The temperature is scheduled during training and controls the convergence to selecting only a single final candidate per optimized layer. The noise-scale adds a random behavior in trying different candidates before deciding for a final one. The state-of-the-art (SotA) does not define rules for setting those important hyperparameters. Therefore, this work elaborates methods to predict feasible start- and end-temperature as well as noise-scale prior to training and to fine-tune the parameters from metrics obtained during a DNAS.