HopScotch: A Holistic Approach to Data Layout-Aware Mapping on NPUs for High-Performance DNN Inference
Suhong Lee, Boyeal Kim, Yongseok Choi, Hyuk‐Jae Lee · ACM Transactions on Architecture and Code Optimization · 2025
Modern deep neural networks (DNNs) are widely utilized across a broad range of domains, scaling rapidly and often comprising hundreds of diverse layers with varying types and configurations. To accelerate DNN execution, specialized hardware solutions, known as neural processing units (NPUs), have been developed. However, this heterogeneity of layers in a DNN model may cause performance degradation on NPUs. For example, while a layer’s execution or dataflow is generally associated with a specific data access order, the data layout in on-chip memory may not be well aligned with it, introducing bubble cycles for layout reordering. Given the hundreds of diverse layers in DNNs, this layout reordering overhead presents a new challenge for achieving efficient end-to-end DNN inference on NPUs. To address this problem, this article introduces HopScotch, a holistic approach to data layout-aware mapping of DNNs on NPUs. First, HopScotch adopts a routing interconnect between the on-chip memory and the systolic array utilizing three-input multiplexers, paired with an on-chip programmable vector processor to manage arbitrary data layout reordering at runtime. Additionally, it introduces a tailored data layout to accommodate a variety of convolutional configurations within the proposed microarchitecture. Second, HopScotch presents a novel layout mapping solver that employs a top-k selection strategy based on a beam search algorithm, facilitating the efficient exploration of the vast layout mapping space at compile time. Third, the proposed layout mapping solver is integrated into the HopScotch mapping framework (HMF) to explore the layout mapping space and evaluate the resulting performance. Experiments with popular DNN models show that HopScotch reduces layout reordering costs by up to 98.2% and 90.3%, resulting in speedups of 2.62× and 1.64× in end-to-end latency, compared to XLA and GCD 2 , respectively.