Generative AI in Multimodal Cross-Lingual Dialogue System for Inclusive Communication Support
Vidhya Nataraj, Wen-Hsuan Liao, Yue‐Shan Chang, Chen-Yu Chiang, Chao‐Yin Lin, Yu-An Lin, Min-Yuh Day · 2024
Advancements in natural language processing have enhanced dialogue systems, making them vital for inclusive technology that facilitates accessible interactions across diverse user needs. However, existing systems often struggle with multimodal inputs, multilingual support, and generating contextually appropriate responses in data-scarce environments. This research addresses these gaps by developing an integrated dialogue system leveraging generative AI models like ChatGPT and multimodal inputs like text, audio, and image. The system utilizes transfer learning and large language models (LLMs) to process multilingual data, generating comprehensive responses tailored to user context. The proposed approach constructs a multimodal cross-lingual task-oriented dialogue system capable of understanding and responding to users in multiple languages and modalities. The proposed multimodal cross-lingual task-oriented dialogue system will enhance functionality and inclusivity compared to traditional unimodal or single-language dialogue systems in providing inclusive communication support. The major research contribution of this study highlights the potential of generative AI in developing accessible dialogue systems that cater to diverse user needs to advance inclusive technology. Practitioner implications of this paper highlight the potential of multimodal cross-lingual dialogue system to foster digital inclusion and inclusive communication support, improving accessibility and equity in human-computer interactions for diverse users.