Gradient-Based Differential Privacy Optimizer for Deep Learning Model Using Collaborative Training Mode
Jing Xia, Weihua Huang, Zhong Ma, Xinfa Dai, He Li · 2019
Deep learning model based on artificial neural network is one of the greatest pushers to realize intelligence of information system unprecedentedly. However, the risk of leaking user data privacy by attacking deep learning model exists in the training process, especially when multi-users concurrently utilize the service of cloud. Motivated by this observation, in order to protect the privacy of deep learning model, this paper proposes a gradient-based differential privacy optimizer using collaborative training mode based on CPU-GPUs hybrid system in cloud. In gradient-based differential privacy optimizer, random sampling, gradient tailoring, gradient-based random perturbation, and advanced privacy budget statistics together guarantee the usability and privacy of the model. Specifically, the effects of privacy budget, noise scale, and privacy deviation parameters and their combinations on accuracy of model are experimentally studied in this paper. To further improve the training efficiency of model, the CPU-GPUs hybrid system is explored as a collaborative training mode for gradient-based differential privacy optimizer. The experimental results indicate that using publicly available dataset MINIST and implemented in TensorFlow is proved to be feasible and efficient. Specifically, our implementation and experiments demonstrate that we can obtain approximately 96% of accuracy under a modest privacy budget. Furthermore, we can achieve up to 20%-30% speed-up ratio in the training process based on gradient-based differential privacy optimizer.