Russian News Similarity Detection with SBERT: pre-training and fine-tuning
SberBank / Moscow, Russia, Aleksei Vatolin, E. Y Smirnova, S. S. Shkarin, SberBank / Moscow, Russia · Computational Linguistics and Intellectual Technologies · 2021
Computation of text similarity is one of the most challenging tasks in NLP as it implies understanding of semantics beyond the meaning of individual words (tokens).Due to the lack of labelled data this task is often accomplished by means of unsupervised methods such as clustering.Within the DE2021: "Russian News Clustering and Headline Selection" we propose a method of building robust text embeddings based on Sentence Transformers architecture, pretrained on a large dataset of in-domain data and then fine-tuned on a small dataset of paraphrases leveraging GlobalMultiheadPooling.