LoBERTa: A Composition Named Entity Recognition Method Based on Longformer and DeBERTa Model
Yida Wang, Yaxuan Zheng, Jiayuan Zhu, Yaqi Yu · 2022
More and more NLP processing models have been proposed, the more popular ones being Longformer and models such as DeBERTa and RoBERTa. The aim of this paper is to propose a better way of processing to recognize elements and text in articles, i.e., automatic segmented text processing. In this paper, the LoBERTa model is proposed to address this limitation. It is a fusion of Longformer and DeBERTa models using a weighted arithmetic average method. And to solve the problem of overfitting, LoBERTa adds a Dropout layer at the head position of the model. Two models, longformer-base-4096 and DeBERTta-large, were trained and then post-processed to place restrictions on the minimum length and minimum confidence of each entity, sieve out predictions smaller than a threshold, and finally perform CV-10Fold and a simple weighted fusion. Three different models were used to classify and evaluate a text dataset of essays written by students in grades 6–12. In the Longformer Baseline model and DeBERTa-large model combined with NER to process the dataset, the experimental results suffer from limitations such as overfitting and low accuracy. The Public LB of the model was 67.8% when using Longformer Baseline and set to 5-Fold, and the Public LB of the DeBERTa-large model alone resulted in 70.5%. The fusion of the two models resulted in 71.2% for LoBERTa, and the final accuracy was 71.8% when trying to repair the labels and replacing 5fold with 10-fold. LoBERTa further improves the accuracy of text processing when compared to using it alone.