SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding
Weihong Xu, Haein Choi, Po-Kai Hsu, Shimeng Yu, Tajana Simunic · ACM Transactions on Embedded Computing Systems · 2025
Large language models (LLMs), composed of Transformer decoders, have demonstrated unparalleled proficiency in understanding and generating human language. However, efficient LLM inference on resource-constraint embedded devices remains a challenge because of the sheer model size and memory-intensive operations that arise from feedforward network (FFN) and multi-head attention (MHA) layers. Existing accelerations offload LLM inference to heterogeneous computing systems comprising expensive memory and processing units. However, recent studies show that most hardware resources are not used because LLM exhibits significant sparsity during inference. The sparsity of LLMs provides a good opportunity to perform memory-efficient inference. In this work, we propose SLIM, an algorithm and hardware co-design optimized for sparse LLM serving on the edge. SLIM exploits LLM’s sparsity by only fetching activated neurons to significantly reduce data movement. To this end, the efficient inference algorithm based on adaptive thresholding is proposed to support runtime configurable sparsity at the cost of negligible accuracy loss. Then, we present the SLIM heterogeneous hardware architecture that combines the best of both near-storage processing (NSP) and processing-in-memory (PIM). SLIM stores FFN weights in high-density 3D NAND and computes FFN layers in NSP units, alleviating high memory requirements caused by FFN weights. The memory-intensive MHA with low arithmetic density is processed in the PIM module. By leveraging the inherent sparsity observed in LLM operations and integrating NSP with PIM techniques within SSDs, SLIM significantly reduces memory footprint, data movement, and energy consumption. Meanwhile, we present the software support for integrating design into existing SSD system. Our comprehensive analysis and system-level optimization demonstrate the effectiveness of our sparsity-tailored accelerator, offering 13-18× throughput improvements over SSD-GPU system and 9-10× better energy efficiency over DRAM-GPU system while maintaining low latency.