Fast Directional Self-Attention Mechanism.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Bo Jiang, Chengqi Zhang · arXiv (Cornell University) · 2018
In this paper, we propose a self-attention mechanism, dubbed self-attention (Fast-DiSA), which is a fast and light extension of directional self-attention (DiSA). The proposed Fast-DiSA performs as expressively as the original DiSA but only uses much less computation time and memory, in which 1) both token2token and source2token dependencies are modeled by a joint compatibility function designed for a hybrid of both dot-product and multi-dim ways; 2) both multi-head and multi-dim attention combined with bi-directional temporal information captured by multiple positional masks are in consideration without heavy time and memory consumption appearing in the DiSA. The experiment results show that the proposed Fast-DiSA can achieve state-of-the-art performance as fast and memory-friendly as CNNs. The code for Fast-DiSA is released at \url{this https URL}.