Feature Selection in Intrusion Detection using Jaccard's Needham Similarity Algorithm
Kamal Kumar, Kartik Gupta, Rohan V. Gupta, Anshul Arora · 2024
The Intrusion Detection System (IDS) plays a vital role in upholding network security, safeguarding network resources and infrastructures. This paper presents an in-depth methodology for enhancing intrusion detection using machine learning in the context of network security. The study integrates various techniques to measure classification performance, with focus on a unique feature selection algorithm called Jaccard-Needham similarity algorithm. This innovative approach quanti-fies dissimilarity between features, strategically deselecting those with the least similarity, thereby enhancing the model's discriminative power. The culmination of our research entails a thorough comparative analysis of machine learning methodologies, encompassing Decision Trees, Random Forest, XGBoost, Gaussian Naive Bayes (GNB), and Logistic Regression. The evaluation process is conducted using the Jaccards feature selection algorithm across different thresholds and is also compared with other existing feature selection techniques. The analysis reveals compelling results with our proposed approach giving highest accuracy (99.41%) at 0.0001 threshold for Jaccard index with top 15 features, outperforming the other evaluated feature selection techniques. This outcome underscores its effectiveness in identifying and selecting features conducive to superior model performance.