RPFNET: Complementary Feature Fusion for Hand Gesture Recognition
Do Yeon Kim, Dae Ha Kim, Byung Cheol Song · 2022 IEEE International Conference on Image Processing (ICIP) · 2022
Hand gesture recognition (HGR) is one of the most challenging tasks because it is very sensitive to occlusion or background. Various modalities such as RGB, depth, and point cloud as well as their combinations have been proposed to improve the performance of HGR, but the fusion of RGB and point cloud with complementary characteristics has never been attempted. This paper analyzes the synergistic effect of the two complementary modalities, and then proposes a new multi-modal fusion network that quantifies and converges the mutual influence of two modalities. Also, to overcome the inherent limitation that the predicted mutual influence does not match the actual one, we propose the self-labeling-based adaptive guidance. Experimental results show that the proposed method achieved 2.46% higher performance than the SOTA method in the case of the NVGesture dataset.