Camera-Radar Fusion With Feature Alignment: Adding Camera Texture to Radar
Xin Jian, Xiaoming Gao, Wanli Dong, Zhilei Zhu · 2024
Radar can enhance target sensing capability after fusion with visible light to achieve all-weather target detection and identification due to lower requirements for weather and light conditions. However, the mainstream radar and camera fusion methods now use decision-level fusion, which fuses the separately processed radar and image data detection results, and fails to take full advantage of the camera's semantic richness and radar's accurate detection distance. Based on this basic observation, we propose a novel feature-level fusion method, which first optimizes for the camera and radar feature misalignment problem by using a deformable attention mechanism to guide the camera features to offset to the corresponding radar positions and then integrates the optimized camera information into two consecutive cross-attention layers, which incorporate the camera and radar features in turn, exploiting the spatial and contextual relationships to achieve stable and efficient fusion. Extensive experimental results on the popular RADIATE dataset have shown the effectiveness of our method. Compared with the baselines, our method performs better under bad weather conditions. Moreover, the proposed method is robust against various real-world scenes such as rain, fog, and snow.