Enhanced Generative Framework With LLMs for Multimodal Emotion-Cause Pair Extraction in Conversations
Xincheng Ju, Dong Zhang, Junhui Li, Shoushan Li, Guodong Zhou · IEEE Transactions on Multimedia · 2025
Emotion-Cause Pair Extraction (ECPE) in conversations aims to identify the emotional utterances (even their categories) along with their corresponding causal utterances, which is crucial in understanding the cause-effect relationship in dialogues. While prior studies of ECPE have predominantly focused on purely textual dialogues and neglected the exploration on the natural scenario of the dialogues with multimodal features, i.e., Multimodal Emotion-Cause Pair Extraction (MECPE) in conversations. To attempt this scenario, we propose a Generative approach for Multimodal Emotion-Cause pair extraction (GMEC) with a single stage, thus effectively reducing errors associated with the propagation and accumulation for MECPE. This approach can not only uniformly handle the information of diverse modalities, but also address all emotion and cause analysis tasks uniformly. Additionally, instead of utilizing the fixed commonsense knowledge base as previously, we resort to the Large Language Models (LLMs), which possess a powerful ability to emerge new knowledge, thereby acting as implicit knowledge engines for MECPE. We refer to this approach as enhanced GMEC. Extensive experimental results and detailed analysis demonstrate a notable improvement in the generative approach. Moreover, the integration of external knowledge from LLMs optimizes the efficiency of data utilization, particularly in few-shot scenarios. The integration of the generative model with LLMs has resulted in a cumulative enhancement of 4.94%, 10.90% on MECPE and MECPE-C (with emotion Category).