LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing
Xiuqing Lv, Peng Zhang, Sunzhu Li, Guobing Gan, Yueheng Sun · 2023
Transformer has become an important technique for natural language processing tasks with great success.However, it usually requires huge storage space and computational cost, making it difficult to be deployed on resourceconstrained edge devices.To compress and accelerate Transformer, we propose LightFormer, which adopts a low-rank factorization initialized by SVD-based weight transfer and parameter sharing.The SVD-based weight transfer can effectively utilize the well-trained Transformer parameter knowledge to speed up the model convergence, and effectively alleviate the lowrank bottleneck problem combined with parameter sharing.We validate this method on machine translation, text summarization, and text classification tasks.Experiments show that on IWSLT'14 De-En and WMT'14 En-De, LightFormer achieves similar performance to the baseline Transformer with 3.8× and 1.8× fewer parameters, and achieves 2.3× speedup and 1.5× speedup respectively, generally outperforming recent light-weight Transformers.