Comparing Human-Labeled and LLM-Generated Semantic Features via Cortical Neural Representation
Boda Xiao, Bo Wang, Xuning Chen, Xiran Xu, WU Xi-hong, Jing Chen · 2024
Understanding how concepts are represented in the brain is a crucial yet unresolved question in language comprehension. Cognitive neuroscience research addresses this issue by linking the semantic features of concepts with neural responses. However, obtaining these semantic features from human raters is time-consuming and labor-intensive, limiting their application in conceptual representation studies. Recently, large language models (LLMs) have demonstrated near-human performance in semantic processing. We evaluated the feasibility of using ChatGPT to automatically generate semantic features by analyzing the correlation of semantic features for 80 concrete nouns obtained from both human raters and ChatGPT raters. Subsequently, we recorded MEG signals from seven participants as they listened to the 80 auditory nouns, and used multivariate pattern analysis (MVPA) to examine the spatiotemporal representations of ChatGPT-derived and human-derived semantic features in MEG source activity. Our results show that the two sets of semantic features are significantly correlated and exhibit high consistency across raters. Furthermore, MVPA results indicate that the semantic features derived from both methods display similar spatiotemporal patterns, which are left-lateralized and spatially overlap with the semantic network, emerging around 200 ms post-stimulus. These findings suggest that LLMs can effectively generate human-like semantic features for use in conceptual representation studies.