A Survey on Attention mechanism in NLP
Nan Zhang, Junyeong Kim · 2023
The Attention mechanism is to screen out a small amount of important information from a large amount of information and focus on this important information while ignoring most of the unimportant information, it is a resource allocation mechanism. The attention mechanism is mainly used in the field of Natural Language Processing to assign attention weight to text, the essence of this is to change from focusing on the whole to focusing on the key parts. This paper starts from the attention mechanism framework, Classification is conducted according to whether we can use the BackPropogation algorithm directly to calculate the gradient, whether the input is global or local participation, and the attention implementation function, which gives a systematic overview of the mainstream models of attention mechanisms. Finally, by combining the application of the attention mechanism in various fields, briefly explains the challenges of the attention mechanism.