Computer vision with spiking neural network
Veís Oudjail · theses.fr (ABES) · 2022
Artificial neural networks (ANN) have become a must-have technique in computer vision, a trend that started during the 2012 ImageNet challenge. However, this success comes with a non-negligible human cost for manual data labeling, very important in model learning, and a high energy cost caused by the need for large computational resources. Spiking Neural Networks (SNN) provide solutions to these problems. It is a particular class of ANNs, close to the biological model, in which neurons communicate asynchronously by representing information through spikes. The learning of SNNs can rely on an unsupervised rule: the STDP. It modulates the synaptic weights according to the local temporal correlations observed between the incoming and outgoing spikes. Different hardware architectures have been designed to exploit the properties of SNNs (asynchrony, sparse and local operation, etc.) in order to design low-power solutions, some of them dividing the cost by several orders of magnitude. SNNs are gaining popularity and there is growing interest in applying them to vision. Recent work shows that SNNs are maturing by being competitive with the state of the art on "simple" image datasets such as MNIST (handwritten numbers) but not on more complex datasets. However, SNNs can potentially stand out from ANNs in video processing. The first reason is that these models incorporate an additional temporal dimension. The second reason is that they lend themselves well to the use of event-driven cameras. They are bio-inspired sensors that perceive temporal contrasts in a scene, in other words, they are sensitive to motion. Each pixel can detect a light variation (positive or negative), which triggers an event. Coupling these cameras to neuromorphic chips allows the creation of totally asynchronous and massively parallelized vision systems. The objective of this thesis is to exploit the capabilities offered by SNNs in video processing. In order to explore the potential offered by SNNs, we are interested in motion analysis and more particularly in motion direction estimation. The goal is to develop a model capable of learning incrementally, without supervision and with few examples, to extract spatiotemporal features. We have therefore performed several studies examining the different points mentioned using synthetic event datasets. We show that the tuning of the SNN parameters is essential for the model to be able to extract useful features. We also show that the model is able to learn incrementally by presenting it with new classes without deteriorating the performance on the mastered classes. Finally, we discuss some limitations, especially on the weight learning, suggesting the possibility of more delay learning, which are still not very well exploited and which could mark a break with ANNs.