Experiments with Contextualized Word Embeddings for Keyphrase Extraction
Cristian Dascalu, Ștefan Trăușan-Matu · 2021
This paper explores the use of contextualized word embeddings for keyphrase extraction, framed as a sequence labeling task. It compares multiple configurations of neural network architectures that build on top of a transformer encoder. The baseline model is obtained by fine-tuning the BERT/SciBERT transformers. Four architectures are presented and evaluated. Each of them is using a different configuration of LSTM and/or multi-head attention modules. We propose a custom loss function and use it to train all architectures with the exception of the baseline. The paper also documents the differences observed when building on top of either the original word embeddings (obtained from training on a language modelling task) or the fine-tuned embeddings of the baseline model. Finally, we present a case study on the behaviour of the attention mechanisms during inference, highlighting the most interesting findings.