A Quad-Core AI Processing Unit for Generative AI in 4nm 5G Smartphone SoC
Chien‐Hung Lin, Jeng-Yun Hsu, Cheng‐Ying Yu, Chia‐Wei Hsu, Yi‐Min Tsai, Kuo-Sheng Wu, Chung-Lun Huang, Meng-Han Hsieh, Tsung-Yao Lin · 2024
This work presents the quad-core AI processing unit (APU) for efficient execution of generative artificial intelligence (GenAI) applications on smartphones. GenAI applications featured with billions of computations and memory accesses are out of tune with smartphone platforms. The APU applies mixed-precision and hybrid-precision operations, inter-core direct data link, zero-overhead layer fusion, and data broadcasting to reduce 50% of DRAM footprint and 90% of DRAM BW. The 4nm 5G smartphone powered by the APU takes less than 1 second to generate an image with optimized stable diffusion (SD) and outputs more than 20 tokens per second with 7B speculative large language model (LLM).