SLICE: Supersense-based Lightweight Interpretable Contextual Embeddings
Cindy Aloui, Carlos Ramisch, Alexis Nasr, Lucie Barque · 2020
Contextualised embeddings such as BERT have become de facto state-of-the-art references in many NLP applications, thanks to their impressive performances.However, their opaqueness makes it hard to interpret their behaviour.SLICE is a hybrid model that combines supersense labels with contextual embeddings.We introduce a weakly supervised method to learn interpretable embeddings from raw corpora and small lists of seed words.Our model is able to represent both a word and its context as embeddings into the same compact space, whose dimensions correspond to interpretable supersenses.We assess the model in a task of supersense tagging for French nouns.The little amount of supervision required makes it particularly well suited for low-resourced scenarios.Thanks to its interpretability, we perform linguistic analyses about the predicted supersenses in terms of input word and context representations.