Retrofitting Structure-aware Transformer Language Model for End Tasks
Hao Fei, Yafeng Ren, Donghong Ji · 2020
We consider retrofitting structure-aware Transformer language model for facilitating end tasks by proposing to exploit syntactic distance to encode both the phrasal constituency and dependency connection into the language model.A middle-layer structural learning strategy is leveraged for structure integration, accomplished with main semantic task training under multi-task learning scheme.Experimental results show that the retrofitted structure-aware Transformer language model achieves improved perplexity, meanwhile inducing accurate syntactic phrases.By performing structure-aware fine-tuning, our model achieves significant improvements for both semantic-and syntactic-dependent tasks.