From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions
David Mareček, Rudolf Rosa · 2019
We inspect the multi-head self-attention in Transformer NMT encoders for three source languages, looking for patterns that could have a syntactic interpretation.In many of the attention heads, we frequently find sequences of consecutive states attending to the same position, which resemble syntactic phrases.We propose a transparent deterministic method of quantifying the amount of syntactic information present in the self-attentions, based on automatically building and evaluating phrasestructure trees from the phrase-like sequences.We compare the resulting trees to existing constituency treebanks, both manually and by computing precision and recall.