Moving object recognition with optical flow-based fusion of thermal and visible videos
Jing Chen, Chao Liang, Yu Lan, Youtian Du · International Conference on Electronic Information Engineering, Big Data, and Computer Technology (EIBDCT 2022) · 2022
Thermal and visible videos can provide complementary information in the moving object detection and recognition. However, most previous approaches focus on the detection and recognition of moving objects from visible videos. In this paper, we present a two-stage approach to moving object recognition by jointly utilizing the thermal and visible videos. In the first stage, we extract the static appearance and the optical flow of moving objects from both sources of videos based on deep networks and generate the bounding box proposals of moving objects. In this stage, two sources of video frames need to be first registered to cover the same scenes with the same resolution. In the second stage, we design a deep network to recognize the categories of the object proposals generated in the first stage and thus obtain the recognition results. Combining the thermal and visible information for recognizing moving objects can improve the performance especially in the low light conditions. To evaluate the proposed approach, we build a thermal-visible video dataset consisting of 200 video pairs. Experimental results demonstrate the effectiveness of the proposed approach.