Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT
Jing Zhao, Yifan Wang, Junwei Bao, Youzheng Wu, Xiaodong He · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2022
Transformer-based pre-trained models, such as BERT, have shown extraordinary success in achieving state-of-the-art results in many natural language processing applications.However, deploying these models can be prohibitively costly, as the standard self-attention mechanism of the Transformer suffers from quadratic computational cost in the input sequence length.To confront this, we propose FCA, a fine-and coarse-granularity hybrid self-attention that reduces the computation cost through progressively shortening the computational sequence length in self-attention.Specifically, FCA conducts an attention-based scoring strategy to determine the informativeness of tokens at each layer.Then, the informative tokens serve as the fine-granularity computing units in selfattention and the uninformative tokens are replaced with one or several clusters as the coarsegranularity computing units in self-attention.Experiments on GLUE and RACE datasets show that BERT with FCA achieves 2x reduction in FLOPs over original BERT with <1% loss in accuracy.We show that FCA offers significantly better trade-off between accuracy and FLOPs compared to prior methods 1 .