A multimodal deep learning approach for recognizing individual actions and group interactions from UAV-based aerial monitoring
Ishrat Zahra, Yanfeng Wu, Hadeel Alsolai, Bayan Ibrahimm Alabdullah, Fatimah Alhayan, Ahmad Jalal, Hui Liu · Kuwait Journal of Science · 2026
Understanding human activities in complex group environments from aerial perspectives represents a critical challenge in unmanned aerial vehicle (UAV)-based surveillance and autonomous systems. We propose a comprehensive multi-modal framework integrating appearance-based and skeletal feature representations for robust group activity recognition. The system employs atmospheric correction, DeepLabv3 segmentation, mask R-CNN detection, and DeepSORT tracking for preprocessing. Feature extraction combines partial differential equation (PDE)-based shape analysis, distance transforms, and heatmap representations with skeletal features including information landscape analysis, manifold projection, and motion signatures. Our novel attention-based fusion methodology optimally integrates these heterogeneous modalities. Spatial relationships are modeled using relational graph convolutional networks (R-GCN) with multi-head attention, while bidirectional long short-term memory (Bi-LSTM) networks capture temporal dependencies. Maximum entropy Markov models (MEMM) enable simultaneous individual and group activity classification. Evaluation on the Okutama-Action UAV dataset achieved 83.8% accuracy for individual actions and 91.6% for group activities, while the JRDB-Act robotics dataset yielded 85.7% and 93.4% accuracy respectively. Our framework demonstrates improvements of 2.3 to 5.0 percentage points over existing UAV-specific methods, achieving competitive performance that advances UAV-specific activity recognition with computational efficiency suitable for offline analysis and batch processing applications, with significant implications for surveillance systems, autonomous robotics, and human behavior analysis applications.