Dependency-Based Self-Attention for Transformer NMT
Hiroyuki Deguchi, Akihiro Tamura, Takashi Ninomiya · 2019
In this paper, we propose a new Transformer neural machine translation (NMT) model that incorporates dependency relations into self-attention on both source and target sides, dependency-based selfattention.The dependency-based selfattention is trained to attend to the modifiee for each token under constraints based on the dependency relations, inspired by linguistically-informed self-attention (LISA).While LISA was originally designed for Transformer encoder for semantic role labeling, this paper extends LISA to Transformer NMT by masking future information on words in the decoderside dependency-based self-attention.Additionally, our dependency-based selfattention operates at subword units created by byte pair encoding.Experiments demonstrate that our model achieved a 1.0 point gain in BLEU over the baseline model on the WAT'18 Asian Scientific Paper Excerpt Corpus Japanese-to-English translation task.