Fine-tuning foundation models for activity of daily living analysis
M Liu · 2023
This thesis presents a novel approach to enhancing the performance of activity recognition systems in daily living scenarios by fine-tuning pretrained foundation models. Recognizing the limitations of traditional small Convolutional Neural Network (CNN) models, which typically process only video inputs and struggle with accuracy and robustness, we explore the integration of large Transformer-based models that utilize both video and audio data. Our methodology involves pretraining on extensive datasets and subsequently fine-tuning on smaller, more specialized datasets. This process allows the model to maintain its general representation capability while improving its adaptability to specific tasks like activity classification, people counting, and user recognition. A key component of our work is the employment of a novel dataset, the UIUC YouHome Activities-of-Daily-Living (ADL) Dataset, featuring diverse activities in varied environmental contexts to ensure a bias-free analysis. This dataset challenges conventional object detection methodologies with its unique light conditions, occlusions, and human-object interactions. We demonstrate that our approach not only surpasses conventional methods in classification accuracy but also exhibits robustness to out-of-distribution data, a crucial aspect for real-world applications. Additionally, our method significantly reduces the computational and time cost of the training process, leveraging the pretrained foundation model's capabilities. However, we acknowledge certain limitations, such as the potential dataset bias in some activities and the sizeable nature of the pretrained models, which might restrict their deployment in smaller devices. Future work will focus on addressing these limitations and further validating the model's robustness with additional datasets, including those with audio input. In conclusion, this thesis contributes to the field of activity recognition in daily living environments by offering a scalable and efficient solution that leverages the strengths of foundation models, setting a new standard for future research and potential applications in this domain.