A Text Generation Model that Maintains the Order of Words, Topics, and Parts of Speech via Their Embedding Representations and Neural Language Models
Noriaki Kawamae · IEEE/WIC/ACM International Conference on Web Intelligence · 2021
Our goal is to generate coherent text accurately in terms of their semantic information and syntactic structure. Embedding methods and neural language models are indispensable in generating coherent text as they learn semantic information, and syntactic structure, respectively, and they are indispensable methods for generating coherent text. We focus here on parts of speech (POS) (e.g. noun, verb, preposition, etc.) so as to enhance these models, and allow us to generate truly coherent text more efficiently than is possible by using any of them in isolation. This leads us to derive Words and Topics and POS 2 Vec (WTP2Vec) as an embedding method, and Structure Aware Unified Language Model (SAUL) as a neural language model. Experiments show that our approach enhances previous models and generates coherent and semantically valid text with natural syntactic structure.