Real-time Web Server Log Processing with Big Data Technologies
Omar A. Alhammadi, Osman Abul · 2024
The continuous monitoring and analysis of system logs are essential for ensuring the stability, security, and per-formance of modern digital systems. However, the sheer volume and complexity of log data pose significant challenges for timely and efficient analysis. In this project, we propose a distributed system for live streaming and analysis of system logs, leveraging the strengths of Kafka, Apache Spark, and Hadoop. Our solution aims to bridge the gap between traditional batch processing approaches and the need for real-time detection and resolution of system issues. By integrating Kafka for reliable log data ingestion, Spark for real-time processing, and Hadoop for distributed storage, we create a scalable pipeline capable of handling large volumes of log data with precision and speed. Through experiments, we demonstrate the scalability and efficiency of our solution, showcasing significant reductions in processing time with increasing numbers of worker nodes and CPU counts. This project lays the groundwork for future endeavors in advanced log analysis and anomaly detection, ultimately enhancing the operational resilience and security posture of organizations in today's digital landscape.