Keyword Extractor for Contrastive Learning of Unsupervised Sentence Embedding
Hua Cai, Weihong Chen, Kehuan Shi, Shuaishuai Li, Qing Xu · 2022
Contrastive learning has been widely applied to learning unsupervised sentence embedding. One major method is unsupervised SimCSE, which only uses random dropouts as noise to build positive pairs for contrastive learning. But token-level matching does not always properly represent the similarity between texts. Sentence-level semantic representation based on keyword-level could enhance the similarity matching performance between texts. To emphasize the contribution of keywords in sentence representation, we propose KESimCSE, which not only includes the constrastive learning of sentence embedding but also adds KL divergence of two elements to the loss function. One element is the dot product of each token embedding and the [CLS] embedding of the BERT outputs, as [CLS] generally captures the semantics of the whole sentence. The other element is the weight list of all tokens obtained through keyword extraction method. Our experiments show that the averaged Spearman’s correlation of KESimCSE on semantic textual similarity tasks raises to 77.21%, which outperforms unsupervised SimCSE by nearly 1.30%.