MSMFNet: Multi-Modal Fusion Gesture Recognition Network with Multi-Scale Integration of AUS and sEMG
Jia-Yu Pan, Jiacheng Chen, Sheng Wei, Jie Pan, Zheng xin Wang · 2024
In the field of gesture recognition, A-mode ultrasound (AUS) and surface electromyography (sEMG) exhibit distinct advantages and limitations. However, they synergistically compensate for each other’s drawbacks, thereby enhancing practical performance. To integrate these modalities effectively, we introduce MSMFNet, a novel multi-scale multi-modal fusion network. This network consists of two key stages: the feature extraction stage and the fusion stage. In the feature extraction stage, we utilize a dual-branch network design that is more suitable for heterogeneous data, allowing effective handling of different types of data and providing a richer feature representation for subsequent steps. Firstly, one branch constructs a multi-scale Conv-Transformer network. Utilizing a combination of convolutional neural networks (CNN) and Transformer, this branch extracts information at various scales from raw sEMG images, comprehensively capturing complex details and providing a global, multi-level feature representation for the task. The second branch utilizes multi-scale mixed convolution for feature extraction from stacked sEMG and AUS images, enabling simultaneous processing of information from two distinct sources. This facilitates the effective fusion of multi-modal information, enhancing the model’s overall understanding of the correlation between different data sources. In the fusion stage, we incorporate an innovative adaptive weight-learning fusion mechanism, departing from conventional fusion approaches. This design is crafted to forestall potential performance deterioration arising from feature conflicts. By dynamically learning weights, we guarantee a precise reflection of the importance of features extracted from diverse branches in the final outcome, thus significantly amplifying the network’s performance. MSMFNet achieves an average accuracy of 93.09% on our mixed dataset, surpassing current state-of-the-art multimodal fusion methods and validating its superiority and robustness.