Policy-based Detection and Blocking System against Abnormal Applications by Analyzing DNS Traffic
Hikaru Ichise, Yong Jin, Katsuyoshi Iida · 2023
Bot-infected computers, which are compounded by botnet communication, conduct botnet-based cyber attacks using various application protocols. When using legitimate applications, a computer mostly performs domain name resolutions via the DNS full-service resolver of the organization network in advance for further communication with the application servers. During the domain name resolution, a DNS full-service resolver at least obtains the DNS NS (Name Server) records, the corresponding glue A records (IP address of the Name Server), and the application specific records, such as MX (Mail Exchange) record in case of mail transmission using Simple Mail Transfer Protocol (SMTP) of the target domain name. On the other hand, bot-infected computers with abnormal applications directly communicate with the application servers without obtaining these DNS records so that direct outbound application traffic will be generated. In this paper, we focus on this kind of direct outbound application traffic and propose a policy-based detection and blocking system against abnormal applications by analyzing DNS traffic. Specifically, the direct outbound application traffic without corresponding domain name resolutions will be detected and blocked as abnormal network traffic from bot-infected computers. We implemented a prototype system and conducted the feature evaluation on the SMTP protocol. The results confirmed that the proposed system worked correctly as designed.