Know Thyself: Validating Knowledge Awareness of LLM-based Persona Agents
Savita Bhat, Ishaan Shukla, Shirish Subhash Karande · 2025
Large Language Models (LLMs) have demonstrated remarkable capability in simulating human behavior, personality, and language.Such synthetic agents with personalities are considered as cost-effective proxies for real users to facilitate crowd-sourcing efforts like annotations, surveys, and A/B testing.Consequently, it is imperative to validate knowledge awareness of these LLM persona agents when they are customized for future usage.Currently, there is no established way for such evaluation and appropriate mitigation.In this work, we propose a generic evaluation approach to validate LLM based persona agents for correctness, relevance, and diversity in the context of self-awareness and domain knowledge.We evaluate the efficacy of this framework using three LLMs ( Llama, GPT-4o, and Gemma) for domains such as air travel, gaming, and fitness.We also experiment with advanced prompting strategies such as ReAct and Reflexion.We find that though GPT-4o and Llama demonstrate comparable performance, they fail some basic consistency checks under certain perturbations.