Applying SoftTriple Loss for Supervised Language Model Fine Tuning
Witold Sosnowski, Anna Wróblewska, Piotr Gawrysiak · Annals of Computer Science and Information Systems · 2022
We introduce a new loss function based on cross entropy and SoftTriple loss, TripleEntropy, to improve classification performance for fine-tuning general knowledge pre-trained language models.This loss function can improve the robust RoBERTa baseline model fine-tuned with cross-entropy loss by about 0.02-2.29 percentage points.Thorough tests on popular datasets using our loss function indicate a steady gain.The fewer samples in the training dataset, the higher gain-thus, for smallsized dataset, it is about 0.71 percentage points, for mediumsized-0.86 percentage points, for large-0.20 percentage points, and for extra-large 0.04 percentage points.