Cyclic Parameter Sharing for Privacy-Preserving Distributed Deep Learning Platforms
Joohyung Jeony, Dohyun Kimz, Joongheon Kim · 2019
As datasets have been increased in sizes, parallelization strategies (i.e., model parallelization and data parallelization) are actively used for distributed deep learning training. In particular, data parallelization which partitions the input samples is commonly used to distributely train deep neural networks. However, if the data are sensitive in terms of security/privacy and the data have constraints on sharing among multiple computing nodes, it is hard to apply these conventional strategies. In this study, we propose a method that sequentially sharing models in cyclic order during training procedures in order to train deep neural networks while preserving the privacy of datasets. In addition, we discuss practical considerations that can be improved in two-folds, i.e., (i) dataset imbalance problem and (ii) efficient usage of computational resources of idle agents.