Dynamic Scene Graph Generation with Unified Temporal Modeling
Sisi You, Bing-Kun Bao · 2024
Dynamic scene graph generation requires understanding the spatial information intra-frame and temporal information between different frames. Existing methods utilize implicit and explicit modeling algorithms to capture temporal information and correlations by designing network architectures or incorporating prior knowledge. However, they exclusively rely on relationship evolution patterns within distinct post-processing modules that are independent of the temporal encoder, leading to solely modifying the results of relationship prediction and difficulty fully harnessing the temporal cues inherent in the video. To address the above challenge, we propose a Unified Temporal Modeling (UTM) that can integrate temporal encoding and temporal correlation modeling. We leverage the relationship evolution patterns to model temporal correlations that can be adapted to capture more relevant temporal cues during temporal encoding. Additionally, our model can be applied to existing image-based scene graph generation methods, extending their capabilities to video tasks. Extensive experiments on the Action Genome dataset demonstrate the robustness of UTM.