Making Sensing Interactive and Descriptive with LLMs: Context Reasoning from Multi-Sensor Data
Kevin Post, Reo Kuchida, Mayowa Olapade, Zhigang Yin, Huber Flores · 2025
Sensor-based Human Activity Recognition (HAR) and context inference using ML/DL (machine and deep learning) models classify activities but offer limited insights into user behaviors and experiences. For instance, IMU sensor data classified may identify "running" or "kicking," but miss broader contexts like "executing an offensive jump in a football match." We present ContextLLM[2], a context-driven solution powered by Large Language Models (LLMs) that transforms sparse, abstract sensor data into detailed, meaningful context descriptions. By using the OPPORTUNITY dataset[1], we show how LLMs aggregate insights from multiple sensors to infer rich context, addressing challenges of data sparsity and fragmentation, and enhancing context-aware applications. As shown in Figure 1, by producing these descriptions using natural language, ContextLLM unlocks new possibilities for making sensing more interactive, e.g., through sensing co-pilot apps.