Scanner hunter
Guowu Xie, Huy Hang, Michalis Faloutsos · 2014
This paper focuses on detecting and studying HTTP scanners, which are malicious entities that explore a website selectively for "opportunities" that can potentially be used for subsequent intrusion attempts. Interestingly, there is practically no prior work on the detection of these entities, which are different from web crawlers or machines performing network-level reconnaissance activities such as port scanning. Detecting HTTP scanners is challenging as they are stealthy and often only probe a few key places on a website, so finding them is a needle-in-the-haystack problem. At the same time, they pose serious risk because they perform the first, exploratory step to provide the seed information that may allow hackers to compromise a website. Our work makes two main contributions. First, we propose Scanner Hunter, arguably the first method to detect HTTP scanners efficiently. The novelty and success of the method lies in the use of community structure, in an appropriately constructed bipartite graph, in order to expose groups of HTTP scanners. The rationale is that the aggregated behavior makes identifying groups of scanners easier than attempting to profile and label IP addresses individually. Scanner Hunter achieves an impressive 96.5% detection precision, which is roughly twice as high as the precision of the Machine Learning-based methods that we use as reference. Second, we provide an extensive study of HTTP scanners in an effort to understand: (a) their spatial and temporal properties, (b) the techniques and tools used by the scanners, and (c) the types of resources they are looking for, which can provides hints as to what the subsequent penetration attempt may target. We use six months worth of web traffic logs collected in 2012 from a University campus, the websites hosted by which received over 1.9 billion requests from 12.8 million IPs. We found that the number of HTTP scanners is non-trivial with roughly 4,000 IPs engaging in this type of activity per week. Our work will hopefully raise the awareness of the community regarding this problem while at the same time provide a promising detection technique that can provide the basis for mitigating the risk posed by HTTP scanners.