A Novel Method on Convolutional Classification for Unifying Multimodal
Liqing Zhang, H.H He, Md Golam Saklain, Hengbin Li, Xiaoan Liu, Fei Deng · 2024
For the current multimodal classification, there is the drawback that can not cope the any combination of modalities. So, this paper proposes a novel unified convolutional classification model for overcoming the disadvantage. We propose the Giant Multi-Scale Dilation Block (GMSDB) for enhancing Spatial Attention Block, including Large Separable Kernel Attention (LSKA) and Multiscale Swelling Attention Block (MSAB), For LSKA, GMSDB can obtain a larger sensory field via an ultra large kernel, and decompose the 2D convolutional kernel of the deep convolutional layer into cascaded horizontal 1D and vertical 1D kernels, which effectively converts the long text and video data into an ultra large vector matrix representation. For MSDA, with multi-head design, the channels of the feature map can be divided into n different heads, and Sliding Window Expansion of Attention (SWDA) can be performed through different dilation rates at different heads. Based on the above model, the semantic information at various scales within the attended sensory field can be aggregated, and the redundancy of the self-attention mechanism can be reduced effectively. In this way, the long-range dependence problem is successfully solved and the ability on adapting to inputs of different scales from different modes is effectively improved.