Interpretable Entity Representations through Large-Scale Typing
Yasumasa Onoe, Greg Durrett · 2020
In standard methodology for natural language processing, entities in text are typically embedded in dense vector spaces with pre-trained models.The embeddings produced this way are effective when fed into downstream models, but they require end-task fine-tuning and are fundamentally difficult to interpret.In this paper, we present an approach to creating entity representations that are human readable and achieve high performance on entity-related tasks out of the box.Our representations are vectors whose values correspond to posterior probabilities over finegrained entity types, indicating the confidence of a typing model's decision that the entity belongs to the corresponding type.We obtain these representations using a fine-grained entity typing model, trained either on supervised ultra-fine entity typing data (Choi et al., 2018) or distantly-supervised examples from Wikipedia.On entity probing tasks involving recognizing entity identity, our embeddings used in parameter-free downstream models achieve competitive performance with ELMoand BERT-based embeddings in trained models.We also show that it is possible to reduce the size of our type set in a learning-based way for particular domains.Finally, we show that these embeddings can be post-hoc modified through a small number of rules to incorporate domain knowledge and improve performance.