Fusion of Hand-crafted and Deep Features for Empathy Prediction
Saurabh Hinduja, Md Taufeeq Uddin, Sk Rahatul Jannat, Astha Sharma, Shaun J. Canavan · 2019
We propose an approach to the OMG-Empathy Challenge for predicting self-annotated, continuous values of valence within the range [-1,1]. We propose the fusion of hand-crafted and deep features, extracted from both actor and listener data, to predict these valence levels. The handcrafted features include image level fusion, facial landmarks, and spectrogram features. Our proposed fusion approach can utilized in multiple parts (i.e. sub-modules), specifically utilized for the generalized track, leading to unique submissions to address the challenge problem. First, both actor and listener images are fused at the image-level. Secondly, facial landmarks from both the actor and listener are fused into one feature vector which is then used to train a random forest for prediction of continuous valence levels. Finally, we use a weighted fusion of the predicted values from both hand-crafted and deep features. We show competitive results on the OMG-Empathy challenge validation set.