Smooth Embedding and Word Sampling Research Based on Transformer Pointer Generation Network
Meiwei Zhang · International Journal of Machine Learning and Computing · 2021
Textual reasoning and abstraction, which both take in a long text and generate a short digest, are widely implemented in the area of Natural Language Processing (NLP).See et al. pioneer the Seq2Seq and Pointer Generation Network structure to address the summarisation task.Later, Transformer model, a successor of Seq2Seq was developed.However, research on the impact of word frequency on textgenerated tasks is not adequate.In this paper, we propose two methods to evaluate the effect of word frequency: Smooth Embedding and Word Sampling.The experiments witness the improvement of Smooth Embedding performance.On the contrary, Word Sampling fails to meet our expectation.It increases the sensitivity of noise, which is a symbol of over-fitting.