Bespoke: Using LLM agents to generate just-in-time interfaces by reasoning about user intent
Palash Nandy, Sigurđur Örn Ađalgeirsson, Anoop K. Sinha, Tanya Kraljic, Mike Cleron, Lei Shi, Angad Singh, Ashish Chaudhary, Ashwin Ganti, Chris Melancon, Shudi Zhang, David Robishaw, Horia Stefan Ciurdar, Justin Secor, Kenneth Aleksander Robertsen, Kirsten Climer, Madison Le, Mathangi Venkatesan, Peggy Chi, Peixin Li · 2024
Large language models (LLMs) have emerged as a powerful tool for creating personalized knowledge experiences for users, often serving as their own interface through text-based chatbots. The interpretation of user intent and generation of output occur implicitly within the model’s architecture. We propose an alternative approach in a system we call Bespoke where the LLM acts as an agent to explicitly reason about user intent, plan, and generate graphical interfaces to fulfill that intent. This approach enables the creation of visually rich interactions that complement chat-based interactions. By employing a step-by-step reasoning process to reduce ambiguity and keep the model on track, we compose interfaces from a toolkit of widgets, providing a designed and tailored user experience. Our early experiment shows that the output interface differs depending on the interpreted intent. In the current version, these interactions are multimodal in the automatic generation of UI; in future versions, this paradigm can be extended to multiple modalities of input and output. This agentive approach moves the interface towards a personalized, bespoke experience with multimodal interaction that adapts to the user’s intentions. See video demonstration here [2].