Mixture of Prompt Experts for Natural Language Inference
Zi’ou Zheng, Xiaodan Zhu · 2024
Reasoning is a central problem in artificial intelligence (AI). Natural language inference (NLI) is a fundamental task that aims to discern the logical relationship between two natural language sentences. Such a capability underlies many real-life applications. One challenge of NLI is that it involves different reasoning requirements and linguistic phenomena. Recently, deep prompt tuning has achieved performance comparable to full-scale fine-tuning. These trained prompts can fit subtasks or data regimes with only a small number of parameters to avoid overfitting. However, one prompt is not adequate for fitting various data regimes. In this paper, we aim to leverage the benefits of deep prompt tuning to craft a more adaptable model for complex natural language inference problems. We propose a novel parameter-efficient approach, named Mixture of Prompt Experts (MOPE), designed to learn various prompts, addressing the inadequacy of one single prompt for tasks comprising multiple sub-tasks with various distributions. This novel approach integrates the mixture of experts with parameter-efficient prompt tuning. Our experimental results show that MOPE achieves better performance on the natural language inference tasks compared to the baseline models.