A Synergistic Large Language Model and Supervised Learning Approach to Zero-Shot and Continual Activity Recognition in Smart Homes
Naoto Takeda, Roberto Legaspi, Yasutaka Nishimura, Kazushi Ikeda, Thomas Plötz, Sonia Chernova · 2024
Sensor-based activity recognition in smart homes can provide residents with applications such as health monitoring and anomaly detection for the elderly. Machine learning models are often used for human activity recognition, but utilizing them from the day residents move in is challenging and usually unreliable due to the lack of labeled data required for training. Recently, large language models (LLMs) such as GPT3.5 and GPT4 have made remarkable progress in natural language processing with zero- and few-shot learning. We leverage the potential of LLMs in activity recognition tasks, focusing on their zero-shot performance. Our key idea is to use an LLM for recognizing activities in the early stages when data may still be scarce, and gradually transition to using a supervised classification model (SCM) as labeled data accumulate. We conducted experiments on two real-world smart home datasets to examine the synergistic effect of using the LLM and SCM in this way. We found that the LLM demonstrated superior zero-shot performance compared to the SCM that required 10 to 20 days to achieve the same performance. Additionally, the LLM achieved greater than 0.80 precision for some activities from the first day of resident occupancy. We also found the improvement in the LLM's performance was marginal even with additional labeled training data collected from residents. However, when used together, the LLM and SCM consistently maintained higher performance, demonstrating the viability of their combination. Our work is the first to use zero-shot LLMs for ambient sensor-based activity classification tasks.