LGAD: Local and Global Attention Distillation for Efficient Semantic Segmentation
Chen Wang, Yafei Qi, Qi Li, Huawen Liu · Frontiers in artificial intelligence and applications · 2024
Efficient semantic segmentation is essential for a wide array of computer vision applications, and knowledge distillation has emerged as a promising methodology for model compression and efficiency. However, we observed that an excess of positive pixels can dilute attention weights, hindering the student model’s learning process. To tackle this significant challenge, we introduce the Local and Global Attention Distillation (LGAD) framework, a pioneering block-based technique that distills both local and global attention. The LGAD framework segments feature maps and output probabilities into well-defined local and global blocks, effectively mitigating the dilution of attention weights. By doing so, it enhances the distinction between positive and negative pixels, particularly amplifying the focus on salient regions within each local and global block. We have conducted comprehensive experiments on three benchmark datasets, Cityscapes, CamVid, and Pascal VOC 2012. The experiment results demonstrate the effectiveness of our proposed LGAD and confirm its superiority over several state-of-the-art distillation methods for semantic segmentation.