SimEval: Investigating the Similarity Obstacle in LLM-based Hardware Code Generation
Mohammad Akyash, Hadi Mardani Kamali · 2025
The increasing use and efficiency of large language models (LLMs) in digital hardware circuit design has started to revolutionize the early stages of integrated circuit (IC) supply chain design and implementation, pushing towards enhanced automation. Despite these advances, hardware circuits' inherent complexity and limited data present significant challenges. Recent studies have begun to explore various attributes of LLM-generated hardware code, including semantics, syntax, fluency, and flexibility. Given that many code generation methodologies rely on fine-tuned LLMs and face constraints due to the limited availability of datasets for hardware designs, this paper investigates the "diversity" of codes generated by LLMs. We introduce SimEval, a comprehensive, multifaceted metric vector designed to assess the similarity of LLM-generated hardware codes at the syntactic, structural, and behavioral levels, from high-level register transfer (RT-level) to synthesized (gate-level) netlists. SimEval uniquely combines sub-tree matching from abstract syntax trees (AST) with structural similarity based on kernel graphs for control flow graphs (CFG). Our experiments focusing on samples from GPT-3.5 datasets and evaluating their similarity using SimEval, highlight the critical role of SimEval in evaluating LLM-based hardware code generators w.r.t. diversity1.