Word Embedding With Zipf’s Context
Lizheng Gao, Gang Zhou, Junyong Luo, Yongzhong Huang · IEEE Access · 2019
Word embeddings generated by neural language models have achieved great success in many NLP tasks. However, neural language models may be difficult to train and time consuming. In this paper, we introduce a simpler but efficient word embedding method based on cooccurrence matrix factorization. Our method drastically reduces the dimensions of the cooccurrence matrix according to the famous Zipf's word frequency law. We observe that if the sampling times of a target word increase to a certain extent, the context of the target word will follow the Zipf's distribution. Enlightened by this, we propose a novel transformation for the cooccurrence matrix. The built cooccurrence matrix is then factorized by PCA. As PCA simply factorizes the cooccurrence matrix linearly and cannot capture the nonlinear relations of features, we construct an autoencoder to further transform the vectors. We compare our method with some well-known neural language models. Our method shows a comparable performance though it is much simpler than the neural language models.