Preliminary Results of An Experiment on Leveraging Large Language Models to Assist Modelers in Interpreting DEVS Natural Language Models
Valdemar Vicente Graciano Neto, Nicholas Keller, Doohwan Kim, Chungman Seo, Priscilla Carbo, Bernard P. Zeigler · 2024
Discrete Event System Specification (DEVS) Natural Language (DNL) implements the DEVS simulation formalism using a natural languagelike notation. However, DNL models can still be complex, involving multiple inputs/outputs, internal/external state transitions, and arbitrary Java code blocks, which steepens the learning curve and reduces the efficiency of junior modelers. Concurrently, Large Language Models (LLMs) like ChatGPT have gained popularity across various domains for their ability to answer specific questions about referenced content. If an LLM tool could reference simulation models written in DNL, it could potentially greatly increase modeler efficiency. To this end, we developed GEM DEVS Chat, a tool designed to assist developers in understanding DEVS models within a simulation project. This paper presents GEM DEVS Chat and reports on an experiment conducted during a Modeling and Simulation course for undergraduate and graduate students. The experiment involved eight students, divided into control and experimental groups. Results indicate that students assisted by the tool understood DEVS models more quickly and accurately.