Engineering Reliable LLM-Based Data Analysis: An Empirical Study of Schema Grounding, Planning, Verification, and SQL Repair
Mevin Jose · Zenodo (CERN European Organization for Nuclear Research) · 2026
Large Language Models (LLMs) can generate executable analytical SQL while still producing incorrect results because of schema, join, aggregation, filtering, dialect, and semantic errors. This paper presents an empirical study of architectural mechanisms for improving the reliability of LLM-generated analytical SQL, focusing on schema grounding, structured planning, AST-based structural verification, and automated SQL repair. We evaluate a multi-stage reliability pipeline on a frozen 500-query benchmark spanning eight business domains over a relational e-commerce warehouse constructed from the Brazilian Olist public dataset. Under the study's row-multiset result-equivalence comparator, the system achieves a 73.40% Result Equivalence Rate (367/500), 31.00% Exact Match, and 100.00% SQL Execution Success, with mean latency of 64.04 seconds. A controlled 100-query component ablation shows that adding AST-based structural verification to unverified planning improves result equivalence from 15.0% to 26.0% (McNemar's p = 0.0192; odds ratio = 3.75). An audit of 101 verifier-triggered repair cases shows that automated repair is a double-edged mechanism: 97/101 post-repair queries were syntactically valid, but 22 previously correct queries were degraded while only 4 incorrect queries were genuinely recovered. A controlled 50-query synthetic perturbation study further shows substantial vulnerability to typographical noise, with 57.1% retention under typo perturbations. These results support conservative, uncertainty-aware structural verification and repair rather than unrestricted automated self-correction. The study is limited to a single relational schema and custom benchmark and does not establish cross-domain or enterprise-wide generalization.