Extracting Syntactic Trees from Transformer Encoder Self-Attentions
David Mareček, Rudolf Rosa · 2018
This is a work in progress about extracting the sentence tree structures from the encoder’s self-attention weights, when translating into another language using the Transformer neural network architecture. We visualize the structures and discuss their characteristics with respect to the existing syntactic theories and annotations.