Improve model-contrastive federated learning by momentum contrast
Xiaoyang Chen, Bokang Li, Weiheng Li · 2023
FedAvg is a classic algorithm for Federated Learning. The MOON algorithm is based on the idea of comparative learning. The model in the previous round is a negative sample, and the global model is a positive sample, which improves the local training of FedAvg. However, in the study of MOON, it was found that since the MOON algorithm compares models rather than pictures, the number of negative samples is difficult to expand, and in traditional comparative learning, the total number of negative samples will significantly affect the effect of comparative learning, when the model in this round is far from the model in the previous round, it may approach the earlier model, thus reducing the efficiency of contrastive learning. So, this paper studied more algorithms related to contrastive learning and federated learning. Finally, a dictionary was built based on the MoCo algorithm to save the previously generated local models. The improved MOON algorithm was named FedCOMO. With the extension of negative samples, the training accuracy on the CIFAR-10 dataset can exceed MOON.