FMC-LLM: Enabling FPGAs for Efficient Batched Decoding of 70B+ LLMs with a Memory-Centric Streaming Architecture
Wenheng Ma, Xinhao Yang, Shulin Zeng, Tengxuan Liu, Libo Shen, Hongyi Wang, Shiyao Li, Jiewen Wang, Yuhan Zhang, Hao Guo, Jintao Li, Ziming Zhang, Zhenhua Zhu, Xuefei Ning, Tsung-Yi Ho, Guohao Dai, Yu Wang · 2025
For large language model (LLM) acceleration, FPGAs face two challenges: insufficient peak computing performance and unacceptable accuracy loss of model compression. This paper proposes FMC-LLM to enable FPGAs for efficient batched decoding of 70B+ LLMs.