SOAP.AI: A Collaborative Tool for Documenting Human Behavior in Videos through Multimodal Generative AI
Qingxiao Zheng, Parisa Rabbani, Yu-Rou Lin, Daan Mansour, Yun Huang · 2024
Large Multimodal Models offer new opportunities for analyzing human activities and social behavior in fields requiring expert knowledge. Their in-context learning and adaptive abilities make customization possible for experts without coding skills. This paper introduces SOAP.AI, a collaborative tool facilitating experts to analyze human behaviors using AI. SOAP.AI is designed to foster a sense of ownership during human-AI collaboration, encouraging task modifications and evaluations to meet diverse goals. For instance, teaching AI to recognize behavioral nuances in autistic individuals could enhance AI's inclusion and value alignment. Our demonstration will engage CSCW researchers and HCI practitioners to discuss the design of collaborative AI systems for behavioral insights generation in various settings, such as medical settings, sports, social media, education, home care, and more.