Multimodal knowledge graph construction for intelligent question answering systems: integrating text, image, and audio data
Xia Yang, Leping Tan · Australian Journal of Electrical & Electronics Engineering · 2025
The rise of intelligent question-answering systems has increased the demand for comprehensive, multimodal knowledge graphs that integrate information from diverse data sources such as text, images, and audio. However, constructing such knowledge graphs poses a significant challenge due to the inherent heterogeneity of different modalities and the large volume of data involved. We propose a multimodal knowledge graph construction framework that combines advanced machine-learning techniques with human guidance to address this challenge. Our approach leverages natural language processing, computer vision, and audio analysis algorithms to extract relevant information from text, images, and audio sources, respectively. This extracted information is then integrated using graph-based representation techniques to build a comprehensive knowledge graph. To ensure the accuracy and quality of the resulting knowledge graph, we also incorporate human feedback and validation at various stages of the construction process. Our proposed framework enables intelligent question-answering systems to access diverse information from text, image, and audio sources, enabling more accurate and comprehensive responses to user queries.