Characterizing and Efficiently Accelerating Multimodal Generation Model Inference
Yejin Lee, Alicia Golden, Anna Sun, Basil Hosmer, Bilge Acun, Can Balioglu, Changhan Wang, Charles David Hernandez, Christian Puhrsch, Daniel Haziza, Driss Guessous, Francisco Massa, Jacob P. Kahn, Jeffrey Wan, Jeremy Reizenstein, Jiaqi Zhai, Joe Isaacson, Joel Schlosser, Juan Pino, Kaushik Ram Sadagopan · IEEE Micro · 2025
Generative artificial intelligence (AI) technology is revolutionizing the computing industry, posing new system design and optimization opportunities. In particular, AI’s ability to understand and respond in multiple modalities comes with significant system resource demands. To sustainably scale generative AI capabilities to billions of users in the world, inference must be fast and efficient. This article pinpoints key system design and optimization opportunities by characterizing a family of emerging multimodal generation models on real systems. Autoregressive token generation is a critical latency performance bottleneck, typically dominated by GPU idle time. In addition to memory-intensive attention across the generative AI models, linear operations constitute significant inference latency due to the feed forward networks in Transformer-based models. We demonstrate that state-of-the-art optimization levers, spanning from applications to system software and hardware, set a 3.88× better baseline.