LinWA:Linear Weights Attention for Time Series Forecasting
Qiang Li, Jiwei Qin, Dezhi Sun, Fei Shi, Daishun Cui, Jiachen Xie · 2024
Transformer has achieved excellent results in the field of time series forecasting. However, some recent studies have pointed out that the existing Transformer models are ineffective in preserving the temporal order information. To solve this problem, we propose a Linear Weighted Attention mechanism (LinWA), a new, simple, and effective attention weights calculation method. Linwa calculates the linear weights and the attention weights, respectively, and combines them to reduce the proportion of the original attention weights to use the order information effectively and improve the forecasting performance. Additionally, we introduce an adaptive parameter adjustment mechanism to adjust the proportion between linear weights and attention weights dynamically. Experimental results on six publicly available datasets show that LinWA preserves temporal order well and improves performance on many Transformer models.