Multi-Modal Behavior Classification Using Audio Visual Data of Vanellus Indicus
Vijini Tennekoon, Devinya Vidanage, Isuru Ariyananda, Sakuni Samar, Samadhi Chathuranga Rathnayake, Nelum Chathuranga Amarasena, Samitha Vidhanarachchi, Devaka Keerthi Weerakoon · 2024
Understanding the behavior of the red-wattled lapwing (Vanellus indicus) is essential for conservation amidst habitat loss and human disturbances. This study presents a comprehensive multi-modal approach that combines video and audio data to distinguish between relaxed and agitated states. Initially, several pre-trained convolutional neural networks (CNN) were assessed for video classification, including VGG16, MobileNetV2, ResNet50, and InceptionV3, with VGG16 achieving the highest accuracy at 92.48%. ResNet50 led in audio classification with 97.03% accuracy, closely followed by VGG16 at 95.31%. In the refined solution, Visual Geometry Group -16 (VGG16) was replaced by InceptionResNetV2, chosen for its superior ability to capture subtle behavioral cues, achieving an accuracy of 87.97%. For audio classification, Yet Another Mobile Network (YAMNet) with Logistic Regression was incorporated to effectively capture nuanced audio patterns, attaining 92.19% accuracy. To enhance interpretability and refine behavior detection, You Only Look Once version 8 (YOLOv8) was used for object detection with a mean Average Precision (mAP) of 0.704, allowing the system to focus more precisely on bird-specific actions. Additionally, Gradient-weighted Class Activation Mapping (Grad-CAM) heatmaps were applied to visualize decision-making regions, further improving model precision. A weighted late fusion technique, with video and audio data weighted at 40% and 60% respectively, was employed, combining the unique strengths of each modality and resulting in highly accurate final classifications. This multi-modal system demonstrates the potential of advanced, explainable artificial intelligence (XAI) techniques in real-time avian behavior analysis, providing valuable insights for wildlife monitoring and conservation through an enhanced understanding of behavioral responses to environmental factors.