SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal · 2018
This report summarizes the results of the Se-mEval 2018 task on machine comprehension using commonsense knowledge.For this machine comprehension task, we created a new corpus, MCScript.It contains a high number of questions that require commonsense knowledge for finding the correct answer.11 teams from 4 different countries participated in this shared task, most of them used neural approaches.The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.