Frequency-aware sparse token vision transformer for medical image segmentation
Junlin Sun, Jie Wang, Jianzhao Liu · Discover Artificial Intelligence · 2026
Abstract Accurate segmentation of organs and lesions in medical images is essential for clinical applications such as diagnosis, prognosis, and treatment planning. Vision transformer (ViT)-based models have demonstrated strong performance in medical image segmentation by modeling long-range dependencies through self-attention mechanisms, but these models still face several challenges, including sensitivity to noise, limited ability to capture fine-grained structures, and high computational overhead caused by dense token representations. To address these limitations, we propose a frequency-aware sparse token vision transformer (FS-ViT) for medical image segmentation. The proposed framework incorporates two key modules: a frequency-aware sparse self-attention (FS-SA) module and a frequency-aware MLP (FA-MLP) module. The FS-SA module first transforms spatial tokens into the frequency domain using Fourier transformation and then calculates self-attention scores to capture robust global representations while reducing sensitivity to spatial noise. In addition, a sparse token mechanism is introduced to eliminate redundant tokens, thus improving computational efficiency by lowering overhead. Additionally, the FA-MLP module is proposed to enhance feature representation by modeling fine-grained structures in the frequency domain by employing Fourier-based feature learning and a learnable query mechanism. The FS-SA and FA-MLP modules are integrated into a U-shaped encoder–decoder ViT architecture. The proposed method is evaluated on five medical image segmentation tasks, and the experimental results demonstrate that FS-ViT achieved superior performance compared with other state-of-the-art segmentation methods, highlighting its effectiveness and generalizability for robust and efficient medical image segmentation.