Model-based Federated Reinforcement Distillation
Sefutsu Ryu, Shinya Takamaeda-Yamazaki · GLOBECOM 2022 - 2022 IEEE Global Communications Conference · 2022
Reinforcement learning (RL) is a framework for learning highly rewarding policies through interactions with the environment. The more the agent knows about the environment, the more easily it learns. Therefore, exploration is often performed using multiple agents. However, information gathered by edge devices is not always available to all devices, including the server. Federated learning (FL) is a framework for collaborative learning while preserving the privacy of the training data. Most of the FL methods exchange the trained model parameters instead of the private data. In contrast, some FL methods incorporate knowledge distillation. These methods are known to be superior to parameter exchange methods in terms of communication efficiency. In this study, we apply the idea of distillation-based FL to RL problems. We highlight some difficulties unique to RL and give a solution to them by introducing an environment model. We call this newly proposed method model-based federated reinforcement distillation (Model-based FRD). As with existing distillation-based FL methods, our proposed method achieves high communication efficiency. Our experimental results show that our method reduces communication costs by approximately 821 times.