Detecting capability evaluate of spider detection techniques
Fan Chunlong, Yu Zhouhua, Xu Lei · 2010
Spider is a program for harvesting internet resources. In order to regulate and monitor accessing behavior of spiders, many Spiders Detection Techniques (SDT) are proposed based on Decision Tree and Bayesian Network etc. At present, the evaluation of these detection techniques mainly rely on manual analysis of web log data. In order to objectively and accurately evaluate the detecting capability of SDT, an Evaluation Method based on Trap technique (EMT) is proposed in this paper. The principles of EMT base on the statistical hypothesis that the users captured by trap obeying binomial distribution theory. The traps layout information and users accessing information are used to calculate evaluating indicators. The evaluating result of experiment indicates that EMT has an accurate and scientific conclusion.