More Efficient Training Strategy to Leverage Neurons in Neural Network
Cheng-Fu Liou, Yi-Cheng Yu · 2024
Comparing the structures of current neural networks and the biological brain, it can be observed that many mechanisms correspond to each other according to their functions. From the perspective of the biological brain, different components have different long-term or short-term memory effects, such as the hippopotamus. However, current neural networks do not pay much attention to this aspect. In this work, we incorporated many hypotheses or theories inspired by the territory of neurobiology to retain the learned knowledge. To start with, we follow the inspiration of the synaptic homeostasis hypothesis (SHY) [1] and add an additional training stage to the training process of the model, which can ensure that only the most important information remains intact and the insignificant synapse can be pruned. In other words, we divide the overall training process into two learning stages: synaptogenesis and synaptic sparsifying. We experimentally demonstrate that our novel learning strategy significantly outperforms the traditional solution in training a sequence of tasks at different times on three public datasets, which supports that the proposed method is more efficient for resource-limited edge devices.