HittER: Hierarchical Transformers for Knowledge Graph Embeddings
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, Yangfeng Ji · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021
This paper examines the challenging problem of learning representations of entities and relations in a complex multi-relational knowledge graph.We propose HittER, a Hierarchical Transformer model to jointly learn Entityrelation composition and Relational contextualization based on a source entity's neighborhood.Our proposed model consists of two different Transformer blocks: the bottom block extracts features of each entity-relation pair in the local neighborhood of the source entity and the top block aggregates the relational information from outputs of the bottom block.We further design a masked entity prediction task to balance information from the relational context and the source entity itself.Experimental results show that HittER achieves new stateof-the-art results on multiple link prediction datasets.We additionally propose a simple approach to integrate HittER into BERT and demonstrate its effectiveness on two Freebase factoid question answering datasets.