TDDEHT: Threat Detection Using Distributed Ensembles of Hoeffding Trees on Streaming Cyber Datasets
Asmah Muallem, Sachin S. Shetty, Liang Hong, Jan Wei Pan · 2018
The use of a well-known state-of-the-art classifier, Hoeffding Trees, is generally proposed in data stream mining (DSM) approaches. Most of these approaches generally address achieving improved accuracy when exceedingly complex drifts are present. Unfortunately, only a few minor DSM approaches have been proposed for anomaly-based Intrusion Detection Systems (IDS). Despite the common relation between anomalies and concept-drift. These approaches also validate with outdated cyber datasets. In this paper, we propose an enhanced IDS ensemble framework of distributed diverse Hoeffding Trees built on Spark Streaming. The pivotal component is an extensible framework to include additional Linear Classifiers and essential IDS components. To validate the efficiency of our approach, we perform several experiments using various up-to-date real-world, synthetic cyber-attack and concept-drift datasets. Our results demonstrate IDS evaluation metrics in the 80-90 percentile and an increase in speed and marginal increase in accuracy and Kappa Statistic, when compared to the current state-of-the-art DSM platform, MOA.