A Web Robot Detection Method Based on Content Classification and Statistics
Guo Wei · 2005
With the widely use of search engines, the impact Web robots have on the Web sites should not be ignored. After analyzing the navigational patterns of Web robots, a new algorithm based on content classification and statistics is proposed. The experiment shows that the new algorithm can detect the unknown robots and unfriendly robots who do not obey the Standard for Robot Exclusion.