$K$-Embeddings: Learning Conceptual Embeddings for Words using Context
Thuy Vu, D. Stott Parker · 2016
We describe a technique for adding contextual distinctions to word embeddings by extending the usual embedding process -into two phases.The first phase resembles existing methods, but also constructs K classifications of concepts.The second phase uses these classifications in developing refined K embeddings for words, namely word K-embeddings.The technique is iterative, scalable, and can be combined with other methods (including Word2Vec) in achieving still more expressive representations.Experimental results show consistently large performance gains on a Semantic-Syntactic Word Relationship test set for different K settings.For example, an overall gain of 20% is recorded at K = 5.In addition, we demonstrate that an iterative process can further tune the embeddings and gain an extra 1% (K = 10 in 3 iterations) on the same benchmark.The examples also show that polysemous concepts are meaningfully embedded in our K different conceptual embeddings for words.