Active Listening: Personalized Question Generation in Open-Domain Social Conversation with User Model Based Prompting
Kevin K. Bowden, Yue Fan, Winson Chen, Cui Wen, Davan Harrison, Xin Eric Wang, Marilyn A. Walker · 2024
Large language models (LLMs) capable of casual conversation have recently become widely available.We hypothesize that users of conversational systems want a more personalized experience, and existing work shows that users are highly receptive to personalized questions (PQs).Question Generation tasks, however, focus on factual questions from textual excerpts.To create a PQ generator, we first identify over 400 real user interests by anonymously aggregating ∼39K user models.We then populate prompt templates with these 400 interests and use an LLM to generate PQs customized to user interests.The result is PerQs, a novel corpus of ∼19K question/answer pairs.We evaluate PerQs at scale in the unique context of the Alexa Prize.Our results show significant positive effects on perceived conversation quality.We then fine-tune, deploy, and evaluate PerQy, a neural model that generates PQs in real-time.When evaluated against several competitive LLM baselines, PerQy produced the most natural and engaging responses.