Building High-throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine
Ariel Keller Rorabaugh, Silvina Caíno‐Lores, Travis Johnston, Michela Taufer · IEEE Transactions on Parallel and Distributed Systems · 2022
Neural networks (NN) are used in high-performance computing and high-throughput analysis to extract knowledge from datasets. Neural architecture search (NAS) automates NN design by generating, training, and analyzing thousands of NNs. However, NAS requires massive computational power for NN training. To address challenges of efficiency and scalability, we proposePENGUIN, a decoupled fitness prediction engine that informs the search without interfering in it.PENGUINuses parametric modeling to predict fitness of NNs. Existing NAS methods and parametric modeling functions can be plugged intoPENGUINto build flexible NAS workflows. Through this decoupling and flexible parametric modeling,PENGUINreduces training costs: it predicts the fitness of NNs, enabling NAS to terminate training NNs early. Early termination increases the number of NNs that fixed compute resources can evaluate, thus giving NAS additional opportunity to find better NNs. We assess the effectiveness of our engine on 6,000 NNs across three diverse benchmark datasets and three state of the art NAS implementations using the Summit supercomputer. Augmenting these NAS implementations withPENGUINcan increase throughput by a factor of 1.6 to 7.1. Furthermore, walltime tests indicate thatPENGUINcan reduce training time by a factor of 2.5 to 5.3.