Image and Video Captioning Using Deep Learning and Natural Language Processing
M.S.R. Naidu, Athrva Kulkarni, Sahil Kadam, Siddhesh Joshi, Nilesh P. Sable, Anuradha Yenkikar · 2024
Deep learning models have been a huge success in image recognition which hence can be used for the purpose of text generation. In the field of imaging science, captioning images and videos is regarded as an intellectually difficult job. Visual Geometry Group (VGG); is a standard deep Convolutional Neural Network (CNN) architecture with multiple layers, specifically focusing on the integration of CNN for image feature extraction. Exploring this underlying method, the use of another model is essential for caption generation. Here the Recurrent Neural Network (RNN) comes in use for caption generation from the extracted features. Models named Long Short-Term Memory (LSTM) based on RNN and Bidirectional encoder representation transformer (BERT) based on Transformers have been prominent in ensuring accurate results. The Flicker8k dataset is used which provides a variety of information useful for model training. By testing validation data along with evaluation metrics, we analyze the effectiveness of different models to create consistent and descriptive headlines. Extending our inquiry to encompass title generation using transformer models, while also exploring learning techniques for real-time title generation and delivery using the Open-CV library available in Python to get the output from the camera and display it on screen. The result shows that the LSTM is the best model for captioning, with an accuracy of 65.07% at the epochs of 300 and the BERT model has an accuracy of 31% at the epochs of 2. The findings of this study not only contribute to advancing subtitle enhancement methodologies but also broaden the potential applications of deep learning techniques in this domain.