PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Daxin Jiang · 2022
Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing.Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs.Nonetheless, existing approaches usually depend on a monoaugmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings.A straightforward solution is resorting to more diverse positives from a multiaugmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field.As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations.PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings.PCL performs peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations.Experiments on STS benchmarks verify the effectiveness of PCL against its competitors in unsupervised sentence embeddings. 1