Anatomically-constrained Near-eye Gaze Tracking and Movement Classification using Transformer
Yishan Zhong, Jingyan Xu, Benoit Louis Marteau, Shaun Q. Y. Tan, May Dongmei Wang · 2025
Gaze tracking is valuable for both virtual reality and clinical applications. In virtual reality, it enhances the user experience by enabling techniques such as foveated rendering, which selectively adjusts rendering quality based on where the user is looking. In clinical settings, automatic gaze tracking can facilitate the diagnosis and treatment of medical conditions such as Attention-Deficit/Hyperactivity Disorder (ADHD), depression, and Benign Paroxysmal Positional Vertigo (BPPV). We propose a novel fixed-head gaze tracking and classification system based on BERT, leveraging the power of Transformer architectures for precise gaze estimation. Our approach models the x/y/z coordinates of the eyeball center and gaze vector while estimating the iris center in x/y coordinates for each frame in a sequence. By incorporating anatomical relevance, the system accounts for the fixed attributes of the eyeball, such as its center and radius, while dynamically capturing changes in iris position and gaze vectors over time. Through static and temporal supervision, we pretrain and finetune our end-to-end model on the TEyeD dataset, achieving an average angular deviation of 2.004° and an F1 score of 0.849 on eye movement classification. The pretrained model is also externally validated using the NVGaze dataset and achieved 8.39° without extensive training. Our transformer-based method circumvents the traditional challenge of fitting an ellipse to the iris or pupil by directly estimating the gaze vector, offering a hybrid solution that combines the strengths of existing gaze tracking techniques.