Advancing Formal Verification: Fine-Tuning LLMs for Translating Natural Language Requirements to CTL Specifications
Rim Zrelli, Henrique Amaral Misson, Maroua Ben Attia, Felipe Göhring de Magalhães, Abdo Shabah, Gabriela Nicolescu · 2024
In the domain of formal verification, translating natural language (NL) requirements into Computation Tree Logic (CTL) specifications presents a notable challenge due to the disparity between human-readable documents and formal specifications. This paper introduces a novel approach that leverages Large Language Models (LLMs) to automate this translation process, thereby enhancing the accuracy and efficiency of formal verification practices. We fine-tune three state-of-the-art LLMs—LLAMA3, Mistral, and Qwen2—with a particular focus on optimizing the Mistral model due to its superior performance. Our methodology is supported by the Natural2CTL dataset, consisting of 2,095 NL requirements and their corresponding CTL specifications. We employ evaluation metrics such as validation loss, accuracy, semantic similarity, and Structural Operator Jaccard Similarity (SOJS) for a comprehensive assessment of model performance. Additionally, a comparative analysis with human translators, trained in CTL logic, underscores the LLMs’ potential to match or even surpass human accuracy in translating NL requirements into formal specifications. Our findings reveal that the fine-tuned Mistral model significantly outperforms the other LLMs and human participants, demonstrating superior accuracy in generating CTL specifications. This study advances the field of formal verification by proposing a scalable solution to the NL-to-CTL translation challenge, setting a new benchmark for the integration of AI tools in complex specification tasks.