Multimodal Persona Based Generation of Comic Dialogs
Harsh Agrawal, Aditya Prasad Mishra, Manish Kumar Gupta, Mausam Mausam · 2023
We focus on the novel problem of persona based dialogue generation for comic strips.Dialogs in comic strips is a unique and unexplored area where every strip contains utterances from various characters with each one building upon the previous utterances and the associated visual scene.Previous works like Di-aloGPT, PersonaGPT and other dialog generation models encode two-party dialogues and do not account for the visual information.To the best of our knowledge we are the first to propose the paradigm of multimodal persona based dialogue generation.We contribute a novel dataset, COMSET, consisting of 54K strips, harvested from 13 popular comics available online.Further, we propose a multimodal persona-based architecture, MPDIALOG, to generate dialogues for the next panel in the strip which decreases the perplexity score by ∼10 points over strong dialogue generation baseline models.We demonstrate that there is still ample opportunity for improvement, highlighting the importance of building stronger dialogue systems that are able to generate persona-consistent dialogues and understand the context through various modalities.