Structural reasoning for image-based social relation recognition
Eduardo V. Sousa, Douglas Guimarães Macharet · Computer Vision and Image Understanding · 2023
Modern societies are composed of complex structures that emerge from the relationships between individuals, and the comprehension of these arrangements has the potential to become a powerful tool for intelligent systems. Current image-based social relation recognition methods isolate specific information from the input to capture essential aspects defining these relationships. However, this is an inaccurate approach since the interaction between all these parts form an intricate structure, which is as valuable as the data each piece carries individually. Consequently, capturing this implicit social structure is essential to achieve the high-level reasoning required to identify relationships adequately. In this work, we propose a novel approach to interpret relationships based on three distinct scopes considering individual, relative, and general information. Additionally, it also takes into account prior knowledge and data dependencies between all these different social perspectives. The Social Knowledge Graph (SKG) is proposed based on these concepts, producing a representation capable of replicating the original social structure. This unique representation is exploited with the Social Graph Network (SGN) by employing specific feature aggregation strategies according to the information embedded into the graph. The performance of the proposed method was evaluated in well-known benchmarks for social relation recognition, achieving a new state-of-the-art. Finally, a deep analysis of the methodology and its main concepts is conducted, delivering results that support our interpretation of social relationships. • A new deep learning framework is designed to perform social relation recognition on images. • A novel conceptual organization describing the structure behind social relationships is proposed. • A new representation is introduced, called Social Knowledge Graph (SKG), which enhances the original structure. • The presented deep model performs relation recognition by exploiting this representation.