Token Level Evaluation and Feature Enhancer for Transformer-based Models
Khanddorj Mendbayar, Masaki Aono · 2021
Most recent state-of-the-art models in the natural language processing (NLP) field such as BERT, ALBERT and XLNet share a common architecture of including an embedding and Transformer encoder and/or decoder layers (aka. Transformer blocks). Here we propose Token Level Evaluation and Feature Enhancer (TLEFE) to be added on any of the Transformer-based models. The TLEFE is supposed to upscale token features for every token in the sentence passed into the system. Through experiments using datasets provided by SemEval-2020 Commonsense Validation and Explanation, we have demonstrated that TLEFE applied model performs better than plain model with negligible additional number of parameters.