What if, Behind the Curtain, There Is Only an LLM? A Holistic Evaluation of TinyLlama-Generated Synthetic Cyber Threat Intelligence
Zuzanna Pietrzak, Krzysztof Mączka, Marcin Niemiec · Electronics · 2025
The generation of synthetic cyber threat intelligence (CTI) has emerged as a significant area of research, particularly regarding the capacity of large language models (LLMs) to produce realistic yet deceptive security content. This study explores both the generative and evaluative aspects of CTI synthesis by employing a custom-developed detection system and publicly accessible LLMs. The evaluation combined automated analysis with a human study involving cybersecurity professionals. The results indicate that even a compact, resource-efficient fine-tuned model can generate highly convincing CTI misinformation capable of deceiving experts and AI-based classifiers. Human participants achieved an average accuracy around 50% in distinguishing between authentic and generated CTI reports. However, the proposed hybrid detection model achieved 98.5% accuracy on the test set and maintained strong generalization with 88.5% accuracy on unseen data. These findings demonstrate both the potential of lightweight models to generate credible CTI narratives and the effectiveness of specialized detection systems in mitigating such threats. The study underscores the growing risk of harmful misinformation in AI-driven CTI and highlights the importance of incorporating robust validation mechanisms within cybersecurity infrastructures to enhance defense resilience.