Detecting Stealthy Malware Using Behavioral Features in Network Traffic
Ting-Fang Yen · 2011
It is clearly in the interest of network administrators to detect hosts within their networks that are infiltrated by stealthy malware. Infected hosts (also called bots) can exfiltrate sensitive data to adversaries, or lie in wait for commands from a bot-master to forward spam, launch denial-of-service attacks, or host phishing sites, for example. Unfortunately, it is difficult to detect such hosts, since their activities are subtle and do not disrupt the network. In this thesis, we hypothesize that malware-infected hosts share characteristics in their network behaviors, which are distinct from those of benign hosts. Our approach works by aggregating “similar ” network traffic involving multiple hosts. We identify key characteristics that capture basic properties of botnet operation, and that can be observed even within coarse network traffic summaries, i.e., flow records. Using network traffic collected at the edge routers of the Carnegie Mellon University campus network, and network traffic generated from real bot instances in virtual machines and honeynets running in the wild, we demonstrate that this approach can reliably detect infected hosts with very few false positives. In addition to identifying relevant behavioral features within hosts ’ network activities, another contribution of this thesis is in developing efficient algorithms for analyzing network traffic. Our algorithms utilize methods from diverse areas, including statistics, data mining, machine learning, and metric embeddings. We also