Enhancing Vision Transformers for Facial Expression Recognition
Ali Mohammad Pazandeh, Emad Fatemizadeh · IEEE Access · 2025
Facial expression recognition (FER) remains a challenging problem due to subtle intra-class variations, strong inter-class similarities, and external factors including pose variations, illumination changes, and occlusions. While vision transformers (ViTs) have demonstrated success in general image classification, their ability to model both global semantic context and localized discriminative facial features remains suboptimal for FER. The enhanced vision transformer architecture for facial expression recognition in this study introduces two important innovations. We first create a two-stream representation that simultaneously employs the class token and global average pooling (GAP) to guarantee a more thorough representation that preserves sensitive spatial details while capturing the overall facial context. Second, we present enhanced conditional positional encoding (ECPE), a sophisticated positional encoding technique that captures both local and global spatial dependencies to improve feature learning, and weighted global average pooling (WGAP), a spatially aware pooling mechanism that dynamically emphasizes discriminative facial regions. We evaluate our framework using three prominent FER benchmarks: FER2013, RAF-DB, and FERPlus, exhibiting consistent performance enhancements over conventional transformer baselines and exceeding many state-of-the-art approaches. Ablation studies demonstrate that each element—two-stream representation, WGAP, and ECPE—independently enhances feature learning and classification accuracy. Our approach improves the capacity of vision transformers to recognize subtle facial expressions by integrating global and local feature learning. The approach we present, although specifically tailored for FER, can also be adapted for other fine-grained classification problems necessitating precise discrimination.