Multi-Task Model Fusion with Mixture of Experts Structure
Jingxuan Zhou, Wenhua Xiao, Dayu Zhang, Bowen Fei, Shuai Zhang, Fuxu Chen · 2023
Model fusion, which aggregates multiple models into a single one, presents significant challenges due to the differences between various task models. To simplify model construction and enhance generalization, we propose a multi-task model construction method that employs a mixture of experts architecture. This method considers each task model as an expert, freezing its network structure, and facilitates model fusion by solely training the gating network and adjusting the output of the expert networks. However, the limited capacity of the gating network complicates the accurate allocation of outputs from each expert. To mitigate this issue, we introduce an auxiliary loss function to guide the gating network in aligning inputs with their corresponding expert networks. We validated our approach on the CIFAR-10 dataset, and the experimental results demonstrate the advantages of our method in terms of overall model accuracy, convergence efficiency, and training accuracy.