Contrastive Preference Learning for Neural Machine Translation
Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, Wenjie Li · 2024
There exists a discrepancy between the tokenlevel objective during training and the overall sequence-level quality that is expected from the model.This discrepancy leads to issues like exposure bias.To align the model with human expectations, sequence-level objectives are often used to fine-tune pre-trained models.In this paper, we introduce a contrastive preference model that enhances the traditional Plackett-Luce model by incorporating an indicator function.Building upon this novel preference model, we propose Contrastive Preference Learning (CPL), which uses offline samples with list-wise preferences to fine-tune a pre-trained model in Neural Machine Translation.Our experiments, conducted on three language pairs, demonstrate that CPL outperforms not only the vanilla Transformer model but also other token-level and sequence-level baselines.Furthermore, the ablation study highlights the essential role of the proposed indicator function in achieving this improvement.