DFF: Distributed Forward-Forward Algorithm for Large-Scale Model in Low-Performance Devices
Qikang Deng, Guang Jin, Yohwan Noh, Subin Yu, DoHoon Lee · 2023
The advancement of Artificial Intelligence(AI) technology, especially with the success of Large Language models(LLMs) like ChatGPT and image generation models such as Stable Diffusion, signifies that AI has entered a new era of large-scale models. However, the rapid development of AI models has brought about the disaster of unlimited growth in model size such as the latest LLM model, PanGu-Sigma, which has reached an astonishing size of 1085B. For ordinary individuals, the enormous hardware costs for these large-scale models make it impossible to train and infer a large model. Fortunately, Geoffrey Everest Hinton has proposed the Forward-Forward(FF) algorithm, which aims to deconstruct a large model into smaller layers and each layer updates its weights through its own backpropagation process, which means that each layer is independent. Therefore, inspired by Geoffrey Hinton’s work, this paper introduces a new Distributed Forward-Forward(DFF) algorithm, which distributes each layer of the Forward-Forward algorithm model to different devices based on ROS2’s publisher-subscriber communication mechanism. Then, the model can be computed across different devices. The Distributed Forward-Forward(DFF) algorithm system in this paper theoretically allows multiple low-performance devices to jointly run large models that a single device is hard to run. Classification model tests on MNIST and CIFAR10 datasets show that the accuracy is 0.9292 and 0.4249 respectively, which has the comparable performance of the pure Forward-Forward algorithm (0.9315 and 0.4310). Our implementation is available at [Github].