Confirmation and Specificity Biases in Large Language Models: An Explorative Study
Daniel E. O’Leary · IEEE Intelligent Systems · 2025
This article explores the presence of confirmation bias and specificity bias in three large language models (LLMs): ChatGPT, Claude, and Gemini. To investigate these biases, I adapted a test originally designed for human subjects to study confirmation bias. Using this test, I analyzed how each LLM responded to an identical prompt regarding a rule for a sequence of numbers. Like human subjects, the LLMs exhibited confirmation bias in their responses and generated solutions prone to “overfitting” the data, leading to specificity bias.