Research on Anti-crawler and Anti-Anti-crawler Technology
Xiaolong Zhang, Yuan Chen, Jiawei Liang, Jiawei Zhang, Tianyang Lu, Chuanqing Wang · 2022
In the era of big data, the extensive application of crawler technology makes the operation of the website difficult, and the anti-crawler technology of the website also emerges as the times require. How to develop anti-anti-crawler technology to effectively capture data according to anti-crawler Technology is of great significance for obtaining data. This paper introduces the basic architecture process and corresponding coping strategies of the commonly used website anti-crawler technology and crawler, analyzes the technical methods of anti-crawler, and puts forward some suggestions for anti-anti-crawler technology.