When is a bishop not like a rook? When it’s like a rabbi! Multi-prototype BERT embeddings for estimating semantic relationships
Gabriella Chronis, Katrin Erk · 2020
This paper investigates contextual language models, which produce token representations, as a resource for lexical semantics at the word or type level.We construct multi-prototype word embeddings from bert-base-uncased (Devlin et al., 2018).These embeddings retain contextual knowledge that is critical for some type-level tasks, while being less cumbersome and less subject to outlier effects than exemplar models.Similarity and relatedness estimation, both type-level tasks, benefit from this contextual knowledge, indicating the context-sensitivity of these processes.BERT's token level knowledge also allows the testing of a type-level hypothesis about lexical abstractness, demonstrating the relationship between token-level behavior and type-level concreteness ratings.Our findings provide important insight into the interpretability of BERT: layer 7 approximates semantic similarity, while the final layer (11) approximates relatedness.