Fusion of visible images and thermal image sequences for automated facial emotion estimation
Hung Son Nguyen, Fan Chen, Kazunori Kotani, Bac Hoai Le · Journal of Multimedia · 2014
The visible image-based approach has long been considered the most powerful approach to facial emotion estimation. However it is illumination dependency. Under uncontrolled operating conditions, estimation accuracy degrades significantly. In this paper, we focus on integrating visible images with thermal image sequences for facial emotion estimation. First, to address limitations of thermal infrared (IR) images, such as being opaque to eyeglasses, we apply thermal Regions of Interest (t-ROIs) to sequences of thermal images. Then, wavelet transform is applied to visible images. Second, features are selected and fused from visible features and thermal features. Third, fusion decision using conventional methods, Principal Component Analysis (PCA) and Eigen-space Method based on class-features (EMC), and our proposed methods, thermal Principal Component Analysis (t-PCA) and norm Eigen-space Method based on class-features (n-EMC), is applied. Applying our suggested methods, experiments on the Kotani Thermal Facial Emotion (KTFE) database show significant improvement, proving its effectiveness.