H$_2$O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
Zhang, Zhenyu, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen · arXiv (Cornell University) · 2023
Hyde-IKV is a dynamic management system for the Key-Value (KV) cache in Small Language Models (SLMs). It tackles the memory bottleneck of long-context inference by intelligently prioritizing and compressing the stored context. Instead of abruptly deleting tokens, Hyde-IKV uses a hybrid strategy of positional decay (gradually reducing the influence of older tokens) and importance-based modulation (preserving tokens with high attention scores or large key vector magnitudes). This non-invasive approach, implemented via hooks into the transformer architecture, significantly reduces GPU memory usage and speeds up inference for long sequences while largely preserving the quality of the generated text.