Automated Evaluation of English Grammar Consistency and Case Coverage Across Commercial Large Language Models
Jonai Nephew, Lucien Rasmussen, Eoin Gallagher · 2024
Language generation systems have achieved significant progress in recent years, enabling the production of coherent and contextually relevant text across diverse applications. However, the complexity of English grammar, with its intricate case assignments and syntactic structures, poses ongoing challenges for such systems in terms of generating consistent and accurate grammatical schemas. The evaluation of ChatGPT and Claude provides a novel comparative analysis that assesses their proficiency in maintaining grammatical consistency and syntactic alignment across varied linguistic phenomena. The study employs a methodical approach, utilizing a comprehensive dataset and advanced statistical techniques to quantify the models' performance in key areas such as sentence structure consistency, grammatical case coverage, and error rates. The findings highlight both the strengths and limitations of current language models, offering insights into their capabilities and informing future developments aimed at enhancing their grammatical sophistication. The research concludes with a discussion of the implications for language model development, emphasizing the potential for integrating rule-based constraints and refining model architectures to achieve more advanced linguistic proficiency.