Video Classification via Relational Feature Encoding Networks
Yao Zhou, Jiamin Ren, Jingyu Li, Litong Feng, Shi Qiu, Ping Luo · 2017
In this paper, we propose a novel Relational Feature Encoding Network for video classification. The proposed network uses a set of relational functions wired on top of a backbone convolutional neural network (ConvNet) to generate multiple complementary feature streams on the fly, which are then combined by an aggregation module to form a video-level representation for recognition. The relational functions compute new relational features by applying element-wise operations or a simple projection to pairs of raw ConvNet features, and thus encode the underlying temporal dynamics and relationship of contextual frames which are critical for recognizing video contents. In this work, we explore a number of design choices for both the relational functions and the aggregation functions, and evaluate the resulting deep model on a number of video classification benchmarks, including the extended Fudan-Columbia Video dataset, UCF101, and Kinetics. Experimental results demonstrate that our model is not only well-suited for action recognition, but also exhibits promising performance for general videos.