Fine-Tuning Pre-Trained Language Models with Gaze Supervision
Shuwen Deng, Paul Prasse, David R. Reich, Tobias Scheffer, Lena A. Jäger · 2024
Human gaze data provide cognitive information that reflect human language comprehension, and has been effectively integrated into a variety of natural language processing (NLP) tasks, demonstrating improved performance over corresponding plain text-based models.In this work, we propose to integrate a gaze module into pre-trained language models (LMs) at the fine-tuning stage to improve their capabilities to learn representations that are grounded in human language processing.This is done by extending the conventional purely text-based fine-tuning objective with an auxiliary loss to exploit cognitive signals.The gaze module is only included during training, retaining compatibility with existing pre-trained LMbased pipelines.We evaluate the proposed approach using two distinct pre-trained LMs on the GLUE benchmark and observe that the proposed model improves performance compared to both standard fine-tuning and traditional text augmentation baselines.Our code is publicly available.1