Detecting trigger-based behaviors in botnet malware
Byeong Ho Kang, Ji-Su Yang, Jaehyun Jason So, Czang Yeob Kim · 2015
Malware often hides malicious behaviors which are triggered when constraints are satisfied. The trigger-based behavior makes malware detection harder, and requires manual analysis. The number of daily submitted malware has been increasing, while the scale of the manual analysis of malware still remains not sufficient. We have been studying the system which analyzes trigger-based behavior in various ways. This paper proposes a detection method of trigger-based behavior by satisfying the trigger condition. Symbolic execution is utilized to invoke the behavior, and multiple execution paths are analyzed to detect malicious activities. We have designed BotMelt, a system representing our approach. BotMelt focuses on analyzing the Botnet malware, especially for the C&C worker. We defined network packet data as symbol, and symbolic executes malware samples to invoke trigger-based behavior. We tested 4 in the wild Botnet malware samples: HwDoor, Bisonal, Keyboy and Plez. BotMelt identified most of the trigger-based behaviors, and our analysts successfully detected malicious activities using BotMelt generated traces. We evaluated on the three criteria: the validness of executed codes, the correctness of the malicious activity detection, and the number of detected behaviors compared with all possible behaviors. Although there are some discussions of our proposed method, we think BotMelt could contribute to the trigger-based behavior analysis.