Analysis of the Capability and Training of Chat Bots in the Generation of Rules for Firewall or Intrusion Detection Systems
Bernardo Louro, Raquel Abreu, Joana Cabral Costa, João B. F. Sequeiros, Pedro R. M. Inácio · 2024
Large Language Models (LLMs) have the potential to aid in closing the knowledge gap in several specific technical areas, such as cybersecurity, by providing a means to translate instructions defined in natural language into specialized system or software specifications (e.g., firewall rules). The work described herein aims at an evaluation of the capability of LLMs to generate rules for firewall and Intrusion Detection Systems (IDSs). A preliminary assessment has shown that widely available chat bots have limited capability to generate correct rules and that caution is needed when using their outputs for the aforementioned objective. This work explores three fine-tuning approaches to address these limitations, each of them with a different objective and achieving distinct success rates. The first approach aimed at testing how well the model was able to use the knowledge obtained from the prompts when the question was structured differently, achieving a success rate of 89%. The second approach aimed at testing how well the model could link the knowledge obtained from two different prompts and reached a success rate of 61%. The final approach aimed at testing if the model could create complex rules by first learning simple rules, achieving a success rate of 79%. It can be concluded that fine-tuning is sufficient to improve chat bots into creating syntactically and technically correct rules for firewalls and IDSs. Results suggest that the development of a specialized model for as many attacks, firewalls and IDSs can indeed be achieved.