SkipW: Resource Adaptable RNN with Strict Upper Computational Limit
Tsiry Mayet, Anne Lambert, Pascal Leguyadec, F. Le Bolzer, François Schnitzler · Zenodo (CERN European Organization for Nuclear Research) · 2021
We introduce Skip-Window, a method to allow recurrent neural networks (RNNs) to trade off accuracy for computational cost during the analysis of a sequence. Similarly to existing approaches, Skip-Window extends existing RNN cells by adding a mechanism to encourage the model to process fewer inputs. Unlike existing approaches, Skip-Window is able to respect a strict computational budget, making this model more suitable for limited hardware like edge devices. We evaluate this approach on four datasets: a human activity recognition task, sequential MNIST, IMDB and adding task. Our results show that Skip-Window is often able to exceed the accuracy of existing approaches for a lower computational cost while strictly limiting said cost.