Dependency-based Mixture Language Models
Zhixian Yang, Xiaojun Wan · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · 2022
Various models have been proposed to incorporate knowledge of syntactic structures into neural language models.However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2.In this paper, we introduce the Dependency-based Mixture Language Models.In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context.We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks. 1 Models External Parameters? External