DFLM: A Dynamic Facial-Language Model Based on CLIP
Yitao Han, Qiming Li · 2024
With the progression of imaging technology and real-time computing, facial expressions have become a kind of input information for human-computer interaction. In recent years, the accuracy of facial expression recognition technology based on single-frame static images has reached a highly usable level. However, in real-world scenarios, facial expressions are dynamic processes, and the accuracy of dynamic facial expression recognition technology still falls short of the levels achieved by traditional static single-frame recognition methods. Therefore, a dynamic facial expression recognition method, DFLM, based on vision-language multimodal contrastive learning is proposed in this paper. The powerful visual representation capability of CLIP is used to extract features and the Temporal Module is used to model the sequential information between continuous images. Furthermore, a Vision-Text Mixer is introduced to enhance the association between visual and textual features. In the text encoding part of the model, learnable prompt methods are introduced to further optimize model performance. Extensive experimental results demonstrate that the DFLM model achieves leading performance on the DFEW, FERV39k, and MAFW datasets.