Identifying and Deactivating Unusable Modalities to Improve Multimodal CNN Robustness for Road Scene Understanding
Robin Condat, Alexandrina Rogozan, Samia Aïnouz, Abdelaziz Bensrhair · 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) · 2022
Multimodal CNNs have become one of the most reliable solutions for many computer vision tasks in recent years. However, in the field of ADAS, the robustness of the latter is essential, in order to guarantee a reliability in road traffic actors detection in case of sensor malfunction or breakdown. A noisy input modality can not only bring no useful information, but also disturb the good functioning of the CNN. Nonetheless, when a modality is unusable, it is preferable to ignore it, which a multimodal CNN can hardly do without a strategy designed for it. In this paper, we propose Noise Augmentation, a data augmentation technique that produce unusable modalities during CNN training, and Modality Activator Model (ModAM), a CNN that preprocess input modalities, identify and deactivate those unusable. The experiments on KITTI object detection benchmark under several degraded conditions show that the combination of our 2 contributions makes multimodal CNNs significantly more robust, whatever the type of noise applied and on any input modality, without lowering their overall performances when all sensors are well functioning.