Data Efficient Transformers for Wearable Sensor Analysis in Centralized and Federated Environments
Jamie McQuire, Paul Watson, Nick Wright, Hugo Hiden, Michael Catt · 2024
Transformers have rapidly become the dominant architecture for analyzing sequential data, utilizing their self-attention mechanism to effectively capture long-term temporal patterns, outperforming recurrent-based methods across various applications. In this paper, we explore the application of transformers to wearable sensor data, focusing on the analysis of human gait, which is often complex and sensitive. We propose two novel frameworks: Data Efficient Sensor Transformer (DesT) for centralized learning and Federated Data Efficient Sensor Transformer (FeDesT) for federated learning (FL) in edge-computing environments. Both frameworks employ knowledge distillation to improve the generalization of transformers, which can be prone to over-fitting due to the limited labeled data available in wearable sensor applications. Experimental results using human gait data collected from uneven and irregular surfaces show that DesT improves the accuracy by 14.8% when compared to existing transformers. FeDesT reduces computational demands on edge devices while outperforming traditional FL methods for transformers. This work demonstrates the potential of transformers for wearable sensor data analysis in both centralized and federated contexts, particularly where privacy and computational efficiency is paramount.