Fusing eye-gaze and speech recognition for tracking in an automatic reading tutor – a step in the right direction?
Morten Højfeldt Rasmussen, Zheng‐Hua Tan · 2013
In this paper we present a novel approach for automatically tracking the reading progress using a combination of eye-gaze tracking and speech recognition.The two are fused by first generating word probabilities based on eye-gaze information and then using these probabilities to augment the language model probabilities during speech recognition.Experimental results on a small dataset show that the tracking error rate of the system using only speech recognition is 34.9% whereas the tracking error rate for the system that incorporates eye-gaze tracking into the speech recognizer is 31.2%-a relative improvement of 10.6%.