Implementing batch normalization-like algorithm with a single spintronic neuron
Zhuo Xu, Yue Xin, Zhengping Yuan, Kang Zhou, Yumeng Yang, Shenghua Gao, Zhifeng Zhu · Chinese Science Bulletin (Chinese Version) · 2024
Spintronic devices have garnered significant attention for their potential in hardware-based implementations of artificial neurons. One of the widely explored mechanisms in this context is the stochastic switching of magnetic tunnel junctions, driven by spin-transfer torque, which is often utilized to produce a sigmoid-shaped activation function. However, a critical limitation in previous studies is that the shape of the activation function remains static throughout the neural network training process. This restricts the ability to adaptively update the weights, leading to inferior performance of the neural network. In this work, we exploit the physics behind the spin torque induced magnetization switching to enable the dynamic change of the activation function during the training process. Specifically, we demonstrate that the pulse width and magnetic anisotropy of the device can be electrically controlled to modify the slope of the activation function, allowing for a faster or slower change in output as required by the backpropagation algorithm. This dynamic control of the activation function is similar to the widely used concept of batch normalization in machine learning, which adjusts and stabilizes the output of each layer during training, thus accelerating the learning process. Thus, in our approach, the ability to adjust the slope of the activation function provides a more flexible hardware-based solution for neural networks. This overcomes the limitations of previous studies where algorithms were predominantly implemented in software and lacked such adaptability in the hardware fields. This work demonstrates that spintronic devices can perform sophisticated tasks like dynamic adjustment of activation functions, marking a significant step forward in hardware-based neural networks. In addition, trainable spintronic neurons can substantially improve the performance of neural networks. For instance, in a handwritten digit recognition task, the recognition accuracy was improved from 88% to 91.3% by employing these dynamically trainable spintronic neurons. This work not only addresses the constraints of fixed activation functions during training but also opens up new possibilities for the integration of machine learning algorithms with spintronic hardware. By utilizing the physical properties of spin-transfer torque and magnetization switching, the proposed spintronic neurons provide a flexible, energy-efficient, and high-performance solution for hardware neural networks. This breakthrough underscores the potential for spintronics to drive the development of next-generation neuromorphic computing systems, offering both performance improvements and significant energy savings in artificial intelligence applications. As spintronics technology continues to evolve, it is poised to become a cornerstone of future AI hardware innovations, enabling more advanced, efficient, and scalable neural computing architectures.