Expression Recognition Based on Visual Transformers with Novel Attentional Fusion
Boju Feng, Hongling Zhang · Journal of Physics Conference Series · 2024
Abstract Facial Expression Recognition (FER) encounters considerable obstacles in practical settings, which are attributed to factors such as occlusions, a range of head positions, facial shape alterations, and motion blur under specific limitations. Despite notable advancements in the automation of FER throughout the past several decades, the majority of prior studies have concentrated on FER within the confines of a laboratory environment. Nonetheless, the complexity of FER is significantly heightened by real-world challenges, including but not limited to occlusions and varied head poses, which lead to areas with insufficient information and intricate backgrounds. As an alternative to earlier CNN-based techniques, we suggest that it is not only viable but also advantageous to convert facial images into a series of visual terms and to tackle expression recognition from a holistic viewpoint. Consequently, we have developed the Visual Transformer with Feature Fusion to confront the challenges of FER in uncontrolled conditions, a process that is divided into two principal stages. Initially, we have put forth the Attention Selective Fusion (ASF) concept to capitalize on the two distinct sets of feature maps produced by a pair of CNN branches. ASF integrates global and local attention mechanisms to synthesize various pieces of discriminative information from these maps. Subsequently, these merged feature maps are transformed into a sequence of visual terms by a process of flattening and projection. Following this, taking cues from the achievements of Transformers in the field of natural language processing, we advocate for the employment of global self-attention to delineate the interrelations among the visual terms. Our proposed strategy has been rigorously tested across three public datasets known for their uncontrolled FER scenarios (RAF-DB, FERPlus, and AffectNet). The results from our comprehensive experiments indicate that our technique surpasses other existing methods, with an accuracy rate of 88.14% on RAF-DB, 88.81% on FERPlus, and 61.85% on AffectNet, thereby establishing new benchmarks in the field. Moreover, the cross-dataset evaluation conducted on the CK+ dataset affirms the robust generalizability of our proposed approach.