Research on Transformer for Natual Language Processing
Wentao Li · 2022
Recently, Natural language processing tasks have been better solved due to the development of deep learning. This paper is going to offer a complete assessment of PTMs for NLP. First, we introduce language representation learning and some latest research. Then we will compare existing PTMs based on different aspects, including pre-train tasks, applications, and structures. Finally, we define a few ability instructions of PTMs for future studies. In short, we not only highlight some essential masking strategies and pre-training tasks but also reveal the efficiency and effectiveness of Transformer-XL on longer documents.