Word embedding revisited: a new representation learning and explicit matrix factorization perspective
Yitan Li, Linli Xu, Fei Tian, Liang Jiang, Zhong Xiao-wei, Enhong Chen · 2015
Recently significant advances have been witnessed in the area of distributed word representations based on neural networks, which are also known as word embeddings. Among the new word embed-ding models, skip-gram negative sampling (SGNS) in the word2vec toolbox has attracted much atten-tion due to its simplicity and effectiveness. Howev-er, the principles of SGNS remain not well under-stood, except for a recent work that explains SGNS as an implicit matrix factorization of the pointwise mutual information (PMI) matrix. In this paper, we provide a new perspective for further understanding SGNS. We point out that SGNS is essentially a rep-resentation learning method, which learns to repre-sent the co-occurrence vector for a word. Based on the representation learning view, SGNS is in fact an explicit matrix factorization (EMF) of the words’ co-occurrence matrix. Furthermore, extended su-pervised word embedding can be established based on our proposed representation learning view. 1