3D Face Tracking and Multi-Scale, Spatio-temporal Analysis of Linguistically Significant Facial Expressions and Head Positions in ASL
Bo Liu, Jingjing Liu, Yu Xiang, Dimitris Metaxas, Carol Neidle · 2014
Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body.This poses a significant challenge for computer-based sign language recognition.Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings-with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine low-and high-level features of differing spatio-temporal scales.This new approach achieves significantly better tracking and recognition results than our previous methods.