How's Your Day Look? The (Un)Expected Sociolinguistic Effects of User Modeling in a Conversational Agent
Libby Ferland, Wilma Koutstaal · 2020
We performed an ex situ Wizard of Oz study of young adults interacting with an idealized digital personal assistant to discuss daily scheduling concerns and stress levels. We further varied rates of "learning" and personalization with the system to test user preferences and changes in participants' linguistic and psychological interactions with an unadapted versus adapted user model, and to determine whether those changes were attributable to acclimatization with the system or to the modeling capabilities, seeking to address 3 research questions: What are the psycholinguistic characteristics of user interactions with a dialogue system designed to act as a scheduling assistant? How does a system's ability to learn about a user and maintain a user model affect these interactions? Are changes in interaction styles uniquely attributable to user modeling ability rather than simply user familiarity or acclimation with the system? We present a linguistic analysis of the results using summary measures generated by a widely used psycholinguistic text analysis tool. Some of the measures seem to present the slightly paradoxical effect of reduced user engagement when a conversational agent explicitly discloses information about its user model to the user. These results suggest that future studies should take care to consider the degree to which the user model is directly exposed to the user. That is, being overly forthcoming about what has been learned about a user may undermine attempts to tailor conversational agents to actively engage and relate to users.