Research on the application of crawler technology in machine learning
Lei Zhang, Zening Chen, Shufeng Yang · Journal of Physics Conference Series · 2021
Abstract With the rapid development of the Internet, the world wide web has become the carrier of a large number of information, and the search engine has entered the public life. In order to help search engines capture related web resources, web crawlers emerge as the times require. Web crawler is a program that automatically extracts web pages. Some malicious web crawlers will not only steal the information of the website, but also cause irreparable damage to the website. Along with it, anti-crawler, the enemy of web crawler, comes into our life. This paper studies the use of machine learning for crawler recognition, so as to make a better anti crawler strategy, use machine recognition instead of manual recognition, reduce the workload of manual recognition of crawler, reduce the loss of human and material resources, and improve the recognition rate of crawler, so as to protect the website information and website equipment.