Query Selector–Efficient transformer with sparse attention
Jacek Klimek, Jacek Klimek, Jakub Klímek, Jakub Klímek, Witold Kraśkiewicz, Mateusz Topolewski · Software Impacts · 2021
We present Query Selector - a sparse attention Transformer algorithm especially efficient for long-term time series forecasting. Time series forecasting is an old and important area of statistical research with vast practical applications to solving real life problems. In recent years, there has been growing interest in applying Deep Learning algorithms to time series forecasting and modeling. However, the best performing Deep Learning algorithm, Transformer, shows some problems with modeling long term time series due to an inherent weakness of this solution i.e. self-attention mechanism. Memory requirements of the canonical Transformer architecture are growing quadratically with sequence length. Query Selector sparse attention Transformer is an attempt to address this problem.