RG PA at SemEval-2021 Task 1: A Contextual Attention-based Model with RoBERTa for Lexical Complexity Prediction
G. Sasibushana Rao, Maochang Li, Xiaolong Hou, Lianxin Jiang, Yang Mo, Jianping Shen · 2021
In this paper we propose a contextual attentionbased model with two-stage fine-tune training using RoBERTa.First, we perform the firststage fine-tune on corpus with RoBERTa, so that the model can learn some prior domain knowledge.Then we get the contextual embedding of context words based on the tokenlevel embedding with the fine-tuned model.And we use Kfold cross-validation to get K models and ensemble them to get the final result.Finally, we attain the 2nd place in the final evaluation phase of sub-task 2 with pearson correlation of 0.8575.