Simple Idea Discovery in a Minimalist LLM Architecture Implementation
Robert Chihaia, Maria Trocan, Florin Leon · Annals of Computer Science and Information Systems · 2025
Large Language Models (LLMs) capture linguistic structure by operating on sequences of sub-word tokens, yet they often display behaviors that suggest an implicit grasp of high-level concepts.This study probes whether such "ideas" are genuinely encoded in LLM representations and, if so, how faithfully and to what extent.We created a deliberately minimalist LLM (encompassing both tokenizer and transformer architecture)designed to expose internal mechanisms with minimal architectural obscurity.Using a carefully curated toy corpora and probing tasks, we trace how semantically related prompts map onto the model's hidden states.Our findings reveal emergent clustering of conceptually similar inputs even in the stripped-down model.These insights advance our understanding of the representational geometry underpinning modern language models and outline a reproducible framework for future mechanistic studies of semantic abstraction.