Modeling Referential Gaze in Task-oriented Settings of Varying Referential Complexity
Özge Alaçam, Eugen Ruppert, Sina Zarrieß, Ganeshan Malhotra, Chris Biemann, Sina Zarrieß · 2022
Referential gaze is a fundamental phenomenon for psycholinguistics and human-human communication.However, modeling referential gaze for real-world scenarios, e.g. for taskoriented communication, is lacking the welldeserved attention from the NLP community.In this paper, we address this challenging issue by proposing a novel multimodal NLP task; namely predicting when the gaze is referential.We further investigate how to model referential gaze and transfer gaze features to adapt to unseen situated settings that target different referential complexities than the training environment.We train (i) a sequential attentionbased LSTM model and (ii) a multivariate transformer encoder architecture to predict whether the gaze is on a referent object.The models are evaluated on the three complexity datasets.The results indicate that the gaze features can be transferred not only among various similar tasks and scenes but also across various complexity levels.Taking the referential complexity of a scene into account is important for successful target prediction using gaze parameters especially when there is not much data for fine-tuning.