Classification of diversified web crawler accesses inspired by biological adaptation
Naomi Kuze, Shu Ishikura, Takeshi Yagi, Daiki Chiba, Masayuki Murata · International Journal of Bio-Inspired Computation · 2021
To discover and prevent attacks, it is necessary to collect data about the attacks using honeypots and to identify malicious accesses from collected data. In this study, we focus on detecting a massive number of crawler accesses, which complicates the detection of malicious accesses. We adapt AntTree, a bio-inspired clustering scheme that is highly scalable and adaptable, for crawler detection. We also designed a feature vector for crawler detection and propose a cluster interpretation method of AntTree. Our results show that the proposed bio-inspired mechanism can detect crawlers with a low false-negative rate, which is an advantage over conventional schemes for detecting various types of crawler.