XAttentionHAR Ensemble: Leveraging Cross-Modal Attention for Enhanced Activity Recognition
Sarita Sahni, Sweta Jain, Sri Khetwat Saritha · International Journal of Pattern Recognition and Artificial Intelligence · 2024
Human Activity Recognition (HAR) is pivotal in ubiquitous computing, offering benefits to human-centric services such as health monitoring, smart homes, and eldercare systems. HAR leverages smartphones, smartwatches, and other wearable devices to collect sensory data annotated with activity labels, which are then used to train machine learning or deep learning models for automatic activity recognition. Effective HAR systems must integrate information from multiple modalities to accurately assist users. This paper introduces the XAttentionHAR model, an innovative cross-modality attention-based ensemble model for HAR. Our approach utilizes a self-attention module to extract features within each modality and an inter-domain cross-attention module to capture and integrate long-term dependencies across domains. The cross-modality attention mechanism enhances the fusion of diverse modalities, enriching the semantic information. We conducted extensive experiments on the WISDM public dataset, which includes accelerometer and gyroscope data from smartwatches and smartphones. Our results demonstrate that XAttentionHAR outperforms other state-of-the-art methods in activity recognition, achieving 98.48% accuracy for smartphone-based HAR and 98.73% accuracy for smartwatch-based HAR, paving the way for improved human-centric services.