Image Captioning With Sentiment for Indonesian
Khumaeni, Masayu Leylia Khodra · 2024
This paper introduces a novel approach to image captioning with sentiment analysis tailored for the Indonesian language, addressing the need for culturally and linguistically appropriate AI tools. Our contribution includes the creation of a unique dataset comprising images annotated with sentiment la-bels in Indonesian, which fills a critical gap in existing resources. The proposed model architecture integrates pretrained Convo-lutional Neural Network (CNN) models, including Inception, Xception, DenseNet, and EfficientNet, as encoders, leveraging their ability to extract intricate visual features. These CNN models are complemented by a Transformer-based decoder that employs Multihead Attention mechanisms to generate captions that accurately reflect both the visual content and the underlying sentiment. This encoder-decoder framework not only improves the efficiency of caption generation but also significantly enhances the model's ability to capture subtle emotional nuances in visual data. Extensive experimentation demonstrates that the Inception-Transformer model outperforms other configurations, achieving the highest BLEU scores of 0.366 for BLEU-l and ROUGE scores of 0.244 for ROUGE-l in the positive sentiment category. For negative sentiment, the model similarly excels with BLEU scores of 0.323 and ROUGE scores of 0.229. This research has practical implications for e-commerce platforms, where the model can be integrated to allow users to upload product images choose their sentiment and receive sentiment-enriched descriptive captions. Such applications can enhance user engagement, improve customer satisfaction, and provide businesses with actionable insights from user-generated visual content in the Indonesian language. Future work will explore expanding the dataset and refining the model to further improve performance and applicability across different domains.