A Comprehensive Survey On Efficient Transformers
Yasser Elouargui, Mahmoud Zyate, Abdellatif Sassioui, Meriyem Chergui, Mohamed El Kamili, Mohammed Ouzzif · 2023
In recent years, there has been substantial attention directed towards the development of proficient Transformers, which display considerable potential in effectively managing extensive sequences in various fields, including but not limited to natural language processing and computer vision. While researchers have conducted survey papers on this field, there remains a notable gap in papers that specifically present and discuss benchmark datasets tailored to evaluate the efficiency and performance of efficient transformers in specific challenges, notably hierarchical reasoning on long-range sequences, question answering on large documents, and key phrase extraction. Moreover, our paper aims to fill this void by offering a clear and comprehensive survey of efficient Transformers. We focus on three main approaches: sparse attention, linearized attention, and low rank attention, all geared towards mitigating the computational and memory challenges associated with traditional Transformers while ensuring consistently high-performance levels. By presenting and analyzing these critical aspects, our paper endeavors to contribute to the broader understanding and advancements in efficient Transformers, paving the way for further research and innovation in this dynamic field.