Auditing for Federated Learning: A Model Elicitation Approach
Yang Liu, Rixing Lou, Jiaheng Wei · 2023
Federated learning provides a promising paradigm for collecting machine learning models from distributed data sources without compromising users’ data privacy. The success of a credible federated learning system builds on the assumption that the decentralized and self-interested users will be willing to participate to share their local models in a trustworthy way. However, without proper incentives, users might simply opt out of the contribution cycle or will be mis-incentivized to contribute spam/false information. This paper introduces solutions to audit the truthful reporting of a local, user-side machine learning model for federated learning. Our results build on the literature of information elicitation but focus on the questions of verifying machine learning models (rather than human predictions). We provide a scoring rule-based framework that verifies and audits reporting of local models such that truthfully reporting is a Bayesian Nash Equilibrium. We study the market implementation, accuracy as well as robustness properties of our proposed solution. We verify the effectiveness of our methods using MNIST and CIFAR-10 datasets. Particularly we show that by reporting low-quality hypotheses, users will receive decreasing scores (rewards, payment, or higher punishment).