A causal framework for explaining the predictions of black-box sequence-to-sequence models
David Alvarez-Melis, Tommi Jaakkola · 2017
We interpret the predictions of any blackbox structured input-structured output model around a specific input-output pair.Our method returns an "explanation" consisting of groups of input-output tokens that are causally related.These dependencies are inferred by querying the black-box model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to select the most relevant components.We focus the general approach on sequence-tosequence problems, adopting a variational autoencoder to yield meaningful input perturbations.We test our method across several NLP sequence generation tasks.