The Impact of Confidence Ratings on User Trust in Large Language Models
Lifei Wang, Natalie Friedman, Chengchao Zhu, Zeshu Zhu, S. Joy Mountford · 2025
This study investigated how displaying AI confidence levels affected user trust and effectiveness in decision-making contexts.Current chatbot interfaces lack transparency in response reliability, which could lead to misguided trust in AI-generated content.We addressed this limitation through a confidence rating interface that visually communicates model certainty and provides prompt improvement suggestions.We conducted a between-subjects study (n=20) comparing a standard chatbot interface with a confidence rating interface that displays three features: 1) confidence rating, 2) confidence factors, and 3) prompt improvement suggestions.Participants completed tasks which could be done in an enterprise setting.These tasks included asking about travel planning suggestions, fact verification about unfamiliar topics, multi-step problem solving for a timezone, and decision making about a stock value.While we didn't reach statistical significance with this small sample size, results showed that the confidence rating interface tended to improve user effectiveness and confidence, particularly in tasks requiring verification or reasoning.Our findings suggest that combining confidence indicators with prompt suggestions could enhance information evaluation when working with AI systems, with implications for enterprise applications where trust is essential.