Analysis of Robot Detection approaches for ethical and unethical robots on Web server log
Mitali Srivastava, Atul Kumar Srivastava, Rakhi Garg, Parimarjan Mishra · International Journal of Advanced Research in Computer Science · 2017
:Due to proliferation of Web robots, it is becoming important to detect robots on commercial and educational websites. Web robots not only consumes a significant proportion of network bandwidth but also a threat for privacy and security. There are various techniques to detect robot from server log i.e. checking robots.txt access, matching keywords in User agent field, calculating browsing speed etc. In this paper, we have implemented and compared theabove three robot detection approaches on real Web server log. Moreover, we have also applied combined approach to detect robots and found that combined approach works better than other three approaches.