Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
Tejas Srinivasan, Yonatan Bisk · 2022
Numerous works have analyzed biases in vision and pre-trained language models individually -however, less attention has been paid to how these biases interact in multimodal settings.This work extends text-based bias analysis methods to investigate multimodal language models, and analyzes intra-and intermodality associations and biases learned by these models.Specifically, we demonstrate that VL-BERT (Su et al., 2020) exhibits gender biases, often preferring to reinforce a stereotype over faithfully describing the visual scene.We demonstrate these findings on a controlled case-study and extend them for a larger set of stereotypically gendered entities.