KD-BERT: A Lightweight Knowledge Distillation Bidirectional Encoder Representations from Transformers for IoT Network Intrusion Detection
Zhong Cao, Xiaohua Liu, Zheng Zhou, Lei Ding, Wenli Shang · IEEE Transactions on Industrial Informatics · 2025
The Internet of Things (IoT) network intrusion detection (NID) has emerged as a powerful technique for effectively detecting attacks and protecting internet safety across a wide range of applications. However, existing NID models often struggle to achieve good detection performance when deployed in resource-restricted environments. To achieve an optimal balance between detection performance and computational cost, we propose a lightweight knowledge distillation bidirectional encoder representations from Transformers (KD-BERT) framework for efficient IoT intrusion detection. Specifically, we first develop a header-payload pairs tokenization method to preprocess the raw data. Then, we distill rich knowledge from a large pre-trained$\text{BERT}_{\text{teacher}}$model into a smaller$\text{BERT}_{\text{student}}$model with fewer training parameters. By leveraging KD-BERT, the$\text{BERT}_{\text{teacher}}$can be compressed into the$\text{BERT}_{\text{student}}$, reducing the number of parameters from 135 to 9 M, while maintaining high detection accuracy. To effectively obtain the essential features, we design a data transformation strategy and fine-tuned the$\text{BERT}_{\text{student}}$model on several labeled IoT NID tasks. Comprehensive experimental results on three public datasets (telemetry datasets, operating systems datasets and network traffic dataset of IoT (TON-IoT), Edge-IIoTset, and IoMT2024) have demonstrated superior performance compared to state-of-the-art methods.