TA-MAMC at SemEval-2021 Task 4: Task-adaptive Pretraining and Multi-head Attention for Abstract Meaning Reading Comprehension
Jing Zhang, Yimeng Zhuang, Yinpei Su · 2021
This paper describes our system used in the SemEval-2021 Task 4 Reading Comprehension of Abstract Meaning, achieving 1st for subtask 1 and 2nd for subtask 2 on the leaderboard.We propose an ensemble of ELECTRAbased models with task-adaptive pretraining and a multi-head attention multiple-choice classifier on top of the pre-trained model.The main contributions of our system are 1) revealing the performance discrepancy of different transformer-based pretraining models on the downstream task, 2) presentation of an efficient method to generate large task-adaptive corpora for pretraining.We also investigated several pretraining strategies and contrastive learning objectives.Our system achieves a test accuracy of 95.11 and 94.89 on subtask 1 and subtask 2 respectively.