Evaluating Attribution Methods using White-Box LSTMs
Yiding Hao · 2020
Interpretability methods for neural networks are difficult to evaluate because we do not understand the black-box models typically used to test them.This paper proposes a framework in which interpretability methods are evaluated using manually constructed networks, which we call white-box networks, whose behavior is understood a priori.We evaluate five methods for producing attribution heatmaps by applying them to white-box LSTM classifiers for tasks based on formal languages.Although our white-box classifiers solve their tasks perfectly and transparently, we find that all five attribution methods fail to produce the expected model explanations.