Parameter-Effective Contrastive Learning and Isotropy of Different Layers For Better Sentence Embeddings
Tianyang Liu, Wenhua Xu, Kaihui Guo · 2023
Contrastive learning, has made significant progress performance in understanding sentence semantics space. Nevertheless, the cost involved in fine-tuning all parameters of pre-trained models is exceedingly high. Simultaneously, the over-parametrization of pre-trained language models with millions of parameters might impede the model’s expressiveness. To alleviate it, we propose a method that combines efficient parameter tuning with contrastive learning, training only a small set of parameters to acquire superior sentence vector representations. Furthermore, to enhance the efficacy of contrastive learning, we propose a simple isotropic loss. Specifically, we use a loss which combining energy learning and hinge loss to enhance the discriminative ability between the original sentence sample and the explicit negation sentence sample, while transforming the embedding space into an isotropic space. We apply the isotropic loss across various encoding layers of pre-trained language models, aiming to elevate uniformity on the basis of alignment, thereby achieving heightened performance. The experimental results show that in a series of semantic similarity tasks, our proposed PECSE framework achieves significant performance improvement compared to the sentence embedding method simCSE.