Dynamic application-layer protocol analysis for network intrusion detection
Holger Dreger, Anja Feldmann, Michael Mai, Vern Paxson, Robin Sommer · MPG.PuRe (Max Planck Society) · 2006
Many network intrusion detection systems (NIDS) rely on protocol-specific analyzers to extract the higher-level semantic context from a traffic stream.To select the correct kind of analysis, traditional systems exclusively depend on well-known port numbers.However, based on our experience, increasingly significant portions of today's traffic are not classifiable by such a scheme.Yet for a NIDS, this traffic is very interesting, as a primary reason for not using a standard port is to evade security and policy enforcement monitoring.In this paper, we discuss the design and implementation of a NIDS extension to perform dynamic application-layer protocol analysis.For each connection, the system first identifies potential protocols in use and then activates appropriate analyzers to verify the decision and extract higher-level semantics.We demonstrate the power of our enhancement with three examples: reliable detection of applications not using their standard ports, payload inspection of FTP data transfers, and detection of IRC-based botnet clients and servers.Prototypes of our system currently run at the border of three large-scale operational networks.Due to its success, the bot-detection is already integrated into a dynamic inline blocking of production traffic at one of the sites.