Unsupervised Sentence-embeddings by Manifold Approximation and Projection
Subhradeep Kayal · 2021
The concept of unsupervised universal sentence encoders has gained traction recently, wherein pre-trained models generate effective task-agnostic fixed-dimensional representations for phrases, sentences and paragraphs.Such methods are of varying complexity, from simple weighted-averages of word vectors to complex language-models based on bidirectional transformers.In this work we propose a novel technique to generate sentenceembeddings in an unsupervised fashion by projecting the sentences onto a fixed-dimensional manifold with the objective of preserving local neighbourhoods in the original space.To delineate such neighbourhoods we experiment with several set-distance metrics, including the recently proposed Word Mover's distance, while the fixed-dimensional projection is achieved by employing a scalable and efficient manifold approximation method rooted in topological data analysis.We test our approach, which we term EMAP or Embeddings by Manifold Approximation and Projection, on six publicly available text-classification datasets of varying size and complexity.Empirical results show that our method consistently performs similar to or better than several alternative state-of-theart approaches.