Recognition of Human Relationships Using Interactions and Gazes through Video Analysis in Surveillance Footage
Matus Tanonwong, Naoya Chiba, Koichi Hashimoto · 2023
In this paper, we hypothesize that human relationships can be estimated from interactions and the frequency and length of mutual gazes between an individual pair in videos, particularly those captured by surveillance cameras at convenience stores, supermarkets, and shopping malls. Recently, there has been significant research progress in human interaction recognition for video surveillance systems. However, mutual gaze detection in surveillance camera’s views still remains a challenge. To verify our hypothesis, we collected a simple mutual gaze dataset at our laboratory and developed a system for mutual gaze detection built on top of a gaze estimator. We then collected a dataset with two types of relationships in the replica room of a convenience store at our laboratory, deployed our developed mutual gaze detector and existing human interaction recognizer, and introduced a human relationship recognition model with a self-attention module. The results suggest that the model pays attention to detected mutual gaze signals and predicted interactions as it learns to classify the human relationships.