Word importance-based similarity of documents metric (WISDM)
Viktor Botev, Kaloyan Marinov, Florian Schäfer · 2017
We present the Word importance-based similarity of documents metric (WISDM), a fast and scalable novel method for document similarity/distance computation for analysis of scientific documents. It is based on recent advancements in the area of word embeddings. WISDM combines learned word vectors together with traditional count-based models for document similarity computation, eventually achieving state-of-the-art performance and precision. The novel method first selects from two text documents those words that carry the most information and forms a word set for each document respectively. Then it relies on an existing word embeddings model to get the vector representations of the selected words. In the final step, it computes the closeness of the two sets of word vector representations, fit into a matrix, using a correlation coefficient. The presented metric was evaluated on three tasks, relevant to the analysis of scientific documents, and three data sets of open access scientific research. The results demonstrate that WISDM achieves significant performance speed-up in comparison to state-of-the-art metrics with a very marginal drop in precision.