Cross-Modal Translation and Alignment of Sensor Events for Layout-Aware Activity Modeling
Weihang You, Zishuai Liu, Fei Dou · 2025
Recognizing Activities of Daily Living (ADLs) from ambient sensor data requires understanding not only what behaviors occur, but how they unfold over time and across space. While recent language-model-based approaches improve semantic abstraction, they lack spatial grounding and cannot represent layout-sensitive activity patterns. This abstraction-reality gap presents a fundamental barrier to real-world deployment, where layout diversity and spatial dynamics are critical. In this work, we introduce a cross-modal framework that models both semantic descriptions and spatial activations of sensor events. By projecting sensor states onto structured floorplan grids and aligning them with natural language narratives via a CLIP encoder, we enable spatially grounded reasoning over activity trajectories. A dual-stream temporal model captures temporal dependencies across both modalities. This design enables trajectory-aware reasoning over complex behaviors. By bridging symbolic abstraction with explicit spatial structure, our approach enhances activity understanding and lays the foundation for layout-adaptive ADL recognition across heterogeneous smart home environments.