Constractive Multilabel Adaptation with Graph-Aware Transformers (CMAGT)
Zakia Labd, Said Bahassine, Khalid Housni · 2025
Multilabel classification is extremely difficult due to label dependencies, class imbalance, and feature space mismatch. We propose Contrastive Multilabel Adaptation with Graph-Aware Transformers (CMAGT), a framework to facilitate feature alignment and label correlation modeling. Our approach integrates supervised contrastive learning, graph neural networks (GNNs) for label embedding, and adaptive sampling via reinforcement learning to improve classification performance on imbalanced data. Through large-scale experiments, we compare two variants: CMAGT_v1, a combination of contrastive learning and GNNs, and CMAGT_v2, which simplifies the architecture with transformers alone. Our results demonstrate that CMAGT_v2 achieves considerable improvements in classification metrics, improving F1-score from 0.0 to 0.44 and reducing hamming loss. We attribute this success to a better training regimen with learning rate scheduling, gradient clipping, and early stopping. The results demonstrate the success of transformer-based multilabel classifiers and indicate that graph-based label embeddings may not be essential for achieving optimal performance. Our subsequent focus will shift towards selfattention mechanisms for modeling label dependencies, which can elevate the applicability of CMAGT to real-world classification tasks.