FlashDecoding++Next: High Throughput LLM Inference With Latency and Memory Optimization
Guohao Dai, Ke Hong, Qiuli Mao, Xiuhong Li, Jiaming Xu, Haofeng Huang, Hongtu Xia, Xuefei Ning, Shengen Yan, Yun Liang, Yu Wang · IEEE Transactions on Computers · 2025
As the Large Language Model (LLM) becomes increasingly important in various domains, the performance of LLM inference is crucial to massive LLM applications. However, centering around the computational efficiency and the memory utilization, the following challenges remain unsolved in achieving high-throughput LLM inference: (1) Synchronous partial softmax update. The softmax operation requires a synchronous update operation among each partial softmax result, leading to ~20% overheads for the attention computation in LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices performing GEMM in LLM inference tends to be flat, leading to under-utilized computation and 50% performance loss after padding zeros in previous designs (e.g., cuBLAS, CUTLASS, etc.). (3) Memory redundancy caused by activations. Dynamic allocation of activations during inference leads to redundant storage of useless variables, bringing 22% more memory consumption.We presentFlashDecoding++Next, a high-throughput inference engine supporting mainstream LLMs and hardware backends. To tackle the above challenges,FlashDecoding++Nextcreatively proposes: (1) Asynchronous softmax with unified maximum.FlashDecoding++Nextintroduces a unified maximum technique for different partial softmax computations to avoid synchronization. Based on this, a fine-grained pipelining is proposed, leading to 1.18× and 1.14× for theprefillanddecodephases in LLM inference, respectively. (2) Flat GEMM optimization with double buffering.FlashDecoding++Nextpoints out that flat GEMMs with different shapes face varied bottlenecks. Then, techniques like double buffering are introduced, resulting in up to 52% speedup for the flat GEMM operation. (3) Buffer reusing and unified memory management.FlashDecoding++Nextreuses the pre-allocated activation buffers throughout the inference process to remove redundancy. Based on that, we unify the management of different types of storage to further exploit the reusing opportunity. The memory optimization enables up to 1.57× longer sequence to be processed.FlashDecoding++Nextdemonstrates remarkable throughput improvement, delivering up to 68.88× higher throughput compared to the HuggingFace [1] implementation. On average,FlashDecoding++Nextachieves 1.25× and 1.46× higher throughput compared to vLLM [2] and TensorRT-LLM [3] on mainstream LLMs.