FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
Ajay Kumar Jaiswal, Bodun Hu, Lu Yin, Yeonju Ro, Tianlong Chen, Shiwei Liu, Aditya Akella · 2024
Autoregressive Large Language Models (e.g., LLaMa, GPTs) are omnipresent achieving remarkable success in language understanding and generation.However, such impressive capability typically comes with a substantial model size, which presents significant challenges for autoregressive token-by-token generation.To mitigate computation overload incurred during generation, several early-exit and layer-dropping strategies have been proposed.Despite some promising success due to the redundancy across LLMs layers on metrics like Rough-L/BLUE, our careful knowledgeintensive evaluation unveils issues such as generation collapse, hallucination, and noticeable performance drop even at the trivial exit ratio of ∼ 10-15% of layers.We attribute these errors primarily to ineffective handling of the KV cache through state copying during early exit.In this work, we observe the saturation of computationally expensive feed-forward blocks of LLM layers and propose FFN-SkipLLM, which is a novel fine-grained skip strategy for autoregressive LLMs.FFN-SkipLLM leverages an input-adaptive feed-forward skipping approach that can skip ∼ 25-30% of FFN blocks of LLMs with marginal change in performance on knowledge-intensive generation tasks without any requirement to handle the KV cache.Our extensive experiments and ablation studies across benchmarks like MT-Bench, Factoid-QA, and variable-length text summarization illustrate how our simple and easy-touse method can facilitate faster autoregressive decoding.