Comparing Automatic and Human Evaluation of Local Explanations for Text Classification
Dong Nguyen · 2018
Text classification models are becoming increasingly complex and opaque, however for many applications it is essential that the models are interpretable.Recently, a variety of approaches have been proposed for generating local explanations.While robust evaluations are needed to drive further progress, so far it is unclear which evaluation approaches are suitable.This paper is a first step towards more robust evaluations of local explanations.We evaluate a variety of local explanation approaches using automatic measures based on word deletion.Furthermore, we show that an evaluation using a crowdsourcing experiment correlates moderately with these automatic measures and that a variety of other factors also impact the human judgements.