Advancements in Multimodal Social Media Post Summarization: Integrating GPT-4 for Enhanced Understanding
Md Jahangir Alam, Ismail Hossain, Sai Puppala, Sajedul Karim Talukder · 2024
The proliferation of social media platforms, such as Facebook, has led to an exponential increase in diverse user-generated content, including text, images, and videos that could be pivotal in public health contexts. This research paper presents an innovative approach to summarizing Facebook posts that incorporate multiple modalities, aiming to generate concise and informative summaries. Utilizing the MT5 architecture for textual analysis and state-of-the-art computer vision for image and video processing, our approach achieves a seamless integration of diverse modalities. Rigorously evaluated against a comprehensive suite of metrics, including ROUGE and BLEU, our model demonstrates notable effectiveness, with GPT-4 achieving impressive scores across various metrics (BLEU-1: 0.82, BLEU-2: 0.68, BLEU-3: 0.59, ROUGE-1: 0.78, ROUGE-2: 0.60, ROUGE-L: 0.73, METEOR: 0.58, CIDEr: 1.75, SPICE: 0.37). These results highlight our methodology's capability to produce succinct and informative summaries, significantly enhancing decision-making processes in public health monitoring and response. This achievement not only underscores the potential of advanced models in handling the complexity of multi-modal content but also sets a new benchmark for future explorations in social media analytics and user experience enhancement.