Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance
Syrielle Montariol, Étienne Simon, Arij Riabi, Djamé Seddah · 2022
We propose our solution to the multimodal semantic role labeling task from the CON-STRAINT'22 workshop.The task aims at classifying entities in memes into classes such as "hero" and "villain".We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and implement three systems to classify the role of the entities.We propose dynamic sampling strategies to tackle the issue of class imbalance.Finally, we perform qualitative analysis on the representations of the entities.* These authors contributed equally. 1 We take each (meme, entity) pair as independent sam-image OCR entity enc pool