FloSIS: a highly scalable network flow capture system for fast retrieval and storage efficiency
Jihyung Lee, Sungryoul Lee, Junghee Lee, Yung Yi, KyoungSoo Park · 2015
Network packet capture performs essential functions in network management such as attack analysis, network troubleshooting, and performance debugging. As the net-work edge bandwidth exceeds 10 Gbps, the demand for scalable packet capture and retrieval is rapidly increasing. However, existing software-based packet capture systems neither provide high performance nor support flow-level indexing for fast query response. This would either pre-vent important packets from being stored or make it too slow to retrieve relevant flows. In this paper, we present FloSIS, a highly scalable, software-based flow storing and indexing system. Flo-SIS is characterized as the following three aspects. First, it exercises full parallelism in multiple CPU cores and disks at all stages of packet processing. Second, it con-structs two-stage flow-level indexes, which helps mini-mize expensive disk access for user queries. It also stores the packets in the same flow at a contiguous disk loca-tion, which maximizes disk read throughput. Third, we optimize storage usage by flow-level content deduplica-tion at real time. Our evaluation shows that FloSIS on a dual octa-core CPU machine with 24 HDDs achieves 30 Gbps of zero-drop performance with real traffic, con-suming only 0.25 % of the space for indexing. 1