Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!
Jack Hessel, Lillian Lee · 2020
Modeling expressive cross-modal interactions seems crucial in multimodal tasks, such as visual question answering.However, sometimes high-performing black-box algorithms turn out to be mostly exploiting unimodal signals in the data.We propose a new diagnostic tool, empirical multimodally-additive function projection (EMAP), for isolating whether or not cross-modal interactions improve performance for a given model on a given task.This function projection modifies model predictions so that cross-modal interactions are eliminated, isolating the additive, unimodal structure.For seven image+text classification tasks (on each of which we set new state-ofthe-art benchmarks), we find that, in many cases, removing cross-modal interactions results in little to no performance degradation.Surprisingly, this holds even when expressive models, with capacity to consider interactions, otherwise outperform less expressive models; thus, performance improvements, even when present, often cannot be attributed to consideration of cross-modal feature interactions.We hence recommend that researchers in multimodal machine learning report the performance not only of unimodal baselines, but also the EMAP of their best-performing model.