PaCE: Unified Multi-modal Dialogue Pre-training with Progressive and Compositional Experts
Yunshui Li, Binyuan Hui, Zhichao Yin, Min Yang, Fei Huang, Yongbin Li · 2023
Perceiving multi-modal information and fulfilling dialogues with humans is a long-term goal of artificial intelligence.Pre-training is commonly regarded as an effective approach for multi-modal dialogue.However, due to the limited availability of multi-modal dialogue data, there is still scarce research on multi-modal dialogue pre-training.Yet another intriguing challenge emerges from the encompassing nature of multi-modal dialogue, which involves various modalities and tasks.Moreover, new forms of tasks may arise at unpredictable points in the future.Hence, it is essential for designed multi-modal dialogue models to possess sufficient flexibility to adapt to such scenarios.This paper proposes PaCE, a unified, structured, compositional multi-modal dialogue pretraining framework.It utilizes a combination of several fundamental experts to accommodate multiple dialogue-related tasks and can be pre-trained using limited dialogue and extensive non-dialogue multi-modal data.Furthermore, we propose a progressive training method where old experts from the past can assist new experts, facilitating the expansion of their capabilities.Experimental results demonstrate that PaCE achieves state-of-the-art results on eight multi-modal dialog benchmarks.