CoLRP: A Contrastive Learning Abstractive Text Summarization Method with ROUGE Penalty
Caidong Tan, Xiao Na Sun · 2023
Contrastive learning can reduce the impact of ex-posure bias associated with training using maximum likelihood estimation, which aims to pull together positive samples to increase the likelihood of high-quality summaries and push away irrelevant negative samples to reduce the likelihood of low-quality summaries. In contrastive learning-based text summarization methods, a standard method for selecting positive and negative samples is randomly selected within a batch. This method can lead to sampling bias to the extent that the consistency of the representation space is compromised. Therefore, we propose a new method to penalize false negatives based on ROUGE metric scores as weights to sample from the dynamic output of the model training process. The method calculates ROUGE metric scores for penalizing false negatives in real-time and can distinguish between positive and negative samples to ensure spatial consistency and alleviate exposure bias. Experimental results on XSum, CNN/DM, and Multi-News datasets show that our approach effectively improves the performance of the latest text summarization pre-training models.