Personalized Fuzzy Federated Prompt Tuning for Re-Identification
Hongwei Zhao, Weishan Zhang, Haoyun Sun, Yikang Zhao, Yuru Liu, Ziyu Wang · 2024
Contrastive Language-Image Pre-training (CLIP) can not be directly applied in re-identification (ReID) tasks because multiple images may be associated with the same ID description in a batch, which is not applicable for instance contrastive learning in CLIP and brings performance degradation. Additionally, ReID data involves sensitive personal information. Data security and privacy protection mechanism is necessary for model training with multi-party ReID data. Federated learning facilitates collaborative training across multiple parties while preserving data privacy. However, when data is non-independent and non-identically distributed (non-IID), the performance of some clients may decrease. To address these challenges, a Personalized Fuzzy Federated Prompt Tuning method (PFFPT) is proposed in this paper for ReID. PFFPT constructs ID-specific learnable prompt tokens at each client, while the global server performs fuzzy federated clustering and dynamic weight aggregation based on local visual feature embeddings. Furthermore, an ID-contrastive Loss is proposed in local client training to tackle the ID-image matching issue. Comprehensive experiments were conducted on two vehicle ReID datasets and eight person ReID datasets. Compared to the traditional FedAVG, PFFPT achieved Rank-1 improvements of more than 10.87 and 10.48 on person ReID and vehicle ReID tasks, respectively, which demonstrates the effectiveness of the proposed method.