LITE-SNN: L everaging I nherent Dynamics to T rain E nergy-Efficient S piking N eural N etworks for Sequential Learning
Nitin Rathi, Kaushik Roy · IEEE Transactions on Cognitive and Developmental Systems · 2024
Spiking Neural Networks (SNNs) are gaining popularity for their promise of low-power machine intelligence on event-driven neuromorphic hardware. SNNs have achieved comparable performance as ANNs on static tasks (image classification) with lower compute energy. In this work, we explore the inherent dynamics of SNNs for sequential tasks like gesture recognition, sentiment analysis, and sequence-to-sequence learning on data from dynamic vision sensors (DVS) and natural language processing (NLP). Sequential data is generally processed with complex RNNs (LSTM/GRU) with explicit feedback connections and internal states to handle the long-term dependencies. The neuron models in SNNs - integrate-and-fire (IF) or leaky-integrate-and-fire (LIF) - have internal states (membrane potential) that can be efficiently leveraged for sequential tasks. The membrane potential in the IF/LIF neuron integrates the incoming current and outputs an event (or spike) when the potential crosses a threshold value. Since SNNs compute with highly sparse spike-based spatio-temporal data, the energy/inference is lower than LSTMs/GRUs. We also show that SNNs require fewer parameters than LSTM/GRU resulting in smaller models and faster inference. We observe the problem of vanishing gradients in vanilla SNNs for longer sequences and implement a convolutional SNN with attention layers to perform sequence-to-sequence learning tasks. The inherent recurrence in SNNs, in addition to the fully parallelized convolutional operations, provide additional mechanisms to model sequential dependencies that lead to better accuracy than convolutional neural networks (CNNs) with ReLU activations. We evaluate SNN on gesture recognition from the IBM DVS dataset, sentiment analysis from the IMDB movie reviews dataset, and German-to-English translation from the Multi30k dataset. The source code is available athttps://github.com/nitin-rathi/LITE-SNN