Evaluating Theory of Mind in Question Answering
Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, Thomas L. Griffiths · 2018
We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs.Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality.We evaluate a number of recent neural models with memory augmentation.We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models' accuracy decreases notably when random sentences are introduced to the tasks at test. 1