Transformers in Time Series Forecasting: A brief Analysis of the Autoformer Transfer Learning Performance
Witesyavwirwa Vianney Kambale, David Krame Kadurha, Ali Deeb, Fadi Al Machot, Taha Bernabia, Kyandoghere Kyamakya · 2023
For the task of long-term time-series forecasting, transformer models have shown great potential in achieving high prediction accuracy. For this reason, numerous transformer architectures designed for time series forecasting have been introduced in the literature. Nevertheless, the issue of a limited amount of training data in certain domains is a real challenge in deep learning. Transfer learning promises to be the solution. However, it is essential to develop transformer-based transfer learning techniques that can lead to the design of Transformer-based Time Series Pre-Trained Models (TSPTMs) that can be used in this situation. This is why, in this paper, we discuss a brief performance analysis of transfer learning techniques applied to the Autoformer, a transformer model designed for time series forecasting, to draw attention to this area. Initial experimental results show potential transfer learning gain. However, given the complexity of transformer models, various transfer learning techniques need to be developed to advance research in this area.