Surprisingly Easy Hard-Attention for Sequence to Sequence Learning
Shiv Shankar, Siddhant Garg, Sunita Sarawagi · 2018
In this paper we show that a simple beam approximation of the joint distribution between attention and output is an easy, accurate, and efficient attention mechanism for sequence to sequence learning.The method combines the advantage of sharp focus in hard attention and the implementation ease of soft attention.On five translation and two morphological inflection tasks we show effortless and consistent gains in BLEU compared to existing attention mechanisms.