Differences in Novice and Expert Evaluations of Generative AI-Produced Definitions
Caroline C. Williams · Proceedings. · 2025
Widely available generative AI products such as ChatGPT are transforming the ways in which students and professionals work and learn, but little is known about how novices and experts perceive, evaluate, and judge generative AI-produced materials.Established research from cognitive science suggests that novices and experts will attend differently to surface and deep structures of content-specific materials.This preliminary study focuses on examining the accuracy and confidence judgements and annotations from experts and novices in the field of Information Science, comparing their evaluations across three different conditions: a ChatGPT produced definition of 'data'; a human crowd-sourced definition of 'data'; and a ChatGPT produced definition of a nonexistent word, 'techno-data.'The paper concludes with next steps in this research study, as well as potential implications for learning, education, and expertise.