Audio-Visual Fusion for Aggression Detection Using Deep Neural Networks
Noussaiba Jaafar, Zied Lachiri · 2019
Multimodal fusion has become a hot topic in computer vision. For surveillance applications detecting unwanted behavior such as aggression is a complex task. The definition of aggression is ambiguous. Hence, it is difficult to describe this high level concept with simple rules or trivial algorithms. Here, aggression can be defined as an action of disturbing others. In this paper, we study aggression in the context of surveillance and its detection from acoustic and visual cues using deep neural networks. We suggest an approach that uses the merging of audio and video modalities with extra information called meta-features. We use an intermediate level which combines the predictions from each modality, and meta-features that have an influence on the fusion process. We show that detecting aggression can be more accurate with the appropriate features and simple deep neural network models. Our experiments prove that our proposed fusion method outperforms the standard fusion strategies such as feature and decision level fusion, and it acquires the best performance.