IMU2CLIP: Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Aparajita Saraf, Amy Bearman, Babak Damavandi · 2023
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP).The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videoswhile preserving the transitivity across these modalities.We introduce several new IMUbased Wearable AI applications such as motionbased media search, or an LM-based multimodal reasoning with motion sensor data -all using text as the grounding platform.In addition, we show that IMU2CLIP significantly improves downstream performances when finetuned for each application, demonstrating its universal usage as a new pre-trained resource.Our code and models will be released publicly.