Enhancing Hybrid CNN-Transformer via Frequency-Based Bridging for Medical Image Segmentation
Zeng Xinyi, Tang Cheng, Zeng Pinxian, Jiaqi Cui, Yan Bin-yu, Wang Peng, Yan Wang · 2024
Hybrid models that leverage both Convolutional Neural Networks (CNNs) and Transformers are gaining traction in medical image segmentation. However, conventional hybrid models often overlook two issues: firstly, the simplistic connections or interactions between CNN and Transformer architecture lead to underutilization of multi-level features, and secondly, the misalignment between global and local information in the spatial domain often hampers effective feature fusion and interaction. In this paper, we propose FreFormer, a hybrid CNN-Transformer architecture that employs frequency domain transform to harmonize multilevel encoded features. Specifically, FreFormer effectively harnesses the multi-level representations from both CNNs and Transformers, ensuring the preservation of global and local contexts. Our key innovation, the Frequency Bridging Transform (FBT) module, addresses feature misalignment by introducing frequency-based mechanisms that cohesively bridge heterogeneous CNN and Transformer layers. This block adeptly transforms multi-level features from spatial domains into a consistent frequency domain, promoting a more harmonious feature fusion. Experiments have confirmed the exceptional performance of Freformer, highlighting the remarkable potential of the FBT module.