Using Machine Learning to Analyze and Detect Anomalies in SELinux Security Policies
Krish Jain, Pranav Kapoor, Joann Sum, Amir Eaman, Esteve Hassan, Elhadi Shakshuki · Procedia Computer Science · 2025
Analysis of Security-Enhanced Linux (SELinux) policies requires extensive manual effort to identify violations and security mis-configurations. Current tools employ mathematical abstractions that, while theoretically sound, produce outputs that practitioners struggle to interpret effectively. Automated approaches using machine learning have shown promise but fail to capture the complex relationships inherent in SELinux policies. Here we present a novel approach combining graph-based policy representation with neural networks to automate SELinux policy analysis. Our approach represents policies as graph structures and transforms these structures into meaningful vector embeddings to learn continuous feature representations that preserve policy neighborhoods and violation patterns. We develop a flexible policy analysis framework that processes these representations through Random Forest, Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP) models to detect violations. Our experimental results demonstrate that this approach achieves 95% accuracy in identifying security violations while maintaining balanced precision and recall metrics, significantly outperforming existing analysis techniques. Through extensive evaluation on synthetic policy datasets derived from production systems, we show that our method effectively captures diverse violation patterns including separation of duty violations, domain transition issues, and unauthorized access paths. Overall, our work presents an efficient approach for automated, interpretable SELinux policy analysis that bridges the gap between theoretical security models and practical policy management.