HYBERT TEAM AT DIALOGUE EVALUATION 2020: TRANSFORMER FOR HYPERNYM EXTRACTION
Maria Trofimova, Michael Arkhipov · Computational Linguistics and Intellectual Technologies · 2020
The report describes the system developed by the HyBert team for the taxonomy enrichment task at Dialog Evaluation 2020 [12]. In this work we investigate the ability of large pre-trained language models to discover hyponym-hypernym relations. We probed state-of-the-art Transformers on the hypernymy task to evaluate implicit hierarchical knowledge captured during self-supervised training. To do so we use a simple distance-based classifier on the representations produced by the Transformer. Furthermore, we examine the performance of supervised approaches with a wide range of different training and embedding strategies. We show that while being a high capacity model, the Transformer is surprisingly hard to train to resolve hypernymy.