Probing for Referential Information in Language Models
Ionut-Teodor Sorodoc, Kristina Gulordava, Gemma Boleda · 2020
Language models keep track of complex linguistic information about the preceding context -including, e.g., syntactic relations in a sentence.We investigate whether they also capture information beneficial for resolving pronominal anaphora in English.We analyze two state of the art models with LSTM and Transformer architectures, respectively, using probe tasks on a coreference annotated corpus.Our hypothesis is that language models will capture grammatical properties of anaphora (such as agreement between a pronoun and its antecedent), but not semantico-referential information (the fact that pronoun and antecedent refer to the same entity).Instead, we find evidence that models capture referential aspects to some extent -though they are still much better at grammar.The Transformer outperforms the LSTM in all analyses, and exhibits in particular better semantico-referential abilities.