IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models
Chenguang Wang, Xiao Liu, Dawn Xiaodong Song · 2022
We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM).Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge.In particular, LMs are able to answer "fill-in-the-blank" questions when given a pre-defined relation category.Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pretrained LMs.We accomplish this by turning pre-trained LMs into zero-shot OIE systems.Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision.For instance, the zeroshot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets. 1 Dylan was in born Minnesota