Building Semantic Cognitive Maps with Text Embedding and Clustering
Rishabh Choudhary, Omar Alsayed, Simona Doboli, Ali A. Minai · 2022 International Joint Conference on Neural Networks (IJCNN) · 2022
Text embedding using vector space models has recently emerged as the leading way to represent text in natural language processing. These embeddings can be at the level of words, sentences, or larger textual units including entire documents. However, sentence embeddings are especially useful because sentences are the most explicitly specified elements of individual thoughts or ideas comprising a document, discussion, or conversation. Analyzing text at the sentence level thus allows access to its fine-grained semantics while preserving the semantic structure that is lost in bag-of-words approaches. Several deep learning-based models such as BERT and USE provide such contextual representations. However, the resulting embeddings are very high-dimensional, and the individual dimensions are not amenable to interpretable labels. Thus, these embeddings define a semantic space but not an explicitly useful cognitive map. In this paper, we show that an adaptive clustering approach applied to the embeddings produced by a neural network-based language model can produce much lower-dimensional, readily interpretable semantic representations, thus creating a usable cognitive map for applications such as semantic tracking and visualization of discussions, or discerning the sequential semantic structure of long documents.