LLM-powered Multimodal Insight Summarization for UX Testing
Kelsey Turbeville, Jennarong Muengtaweepongsa, Samuel Stevens, Jason D. Moss, Amy Pon, K. Y. Lee, Charu Mehra, J. Villalobos, Ranjitha S. Kumar · 2024
User experience (UX) testing platforms capture many data types related to user feedback and behavior, including clickstream, survey responses, screen recordings of participants performing tasks, and participants’ think-aloud audio. Analyzing these multimodal data channels to extract insights remains a time-consuming, manual process for UX researchers. This paper presents a large language model (LLM) approach for generating insights from multimodal UX testing data. By unifying verbal, behavioral, and design data streams into a novel natural language representation, we construct LLM prompts that generate insights combining information across all data types. Each insight can be traced back to behavioral and verbal evidence, allowing users to quickly verify accuracy. We evaluate LLM-generated insight summaries by deploying them in a popular remote UX testing platform, and present evidence that they help UX researchers more efficiently identify key findings from UX tests.