Optimization and application of web crawler architecture
Kuncheng Li, Junqi Fei, Chunmei Fan · Third International Conference on Computer Science and Communication Technology (ICCSCT 2022) · 2022
For monitoring of cutting-edge technologies by obtaining the massive data on the internet, the pyspider framework is used to regularly crawl the information of a large number of websites, and the crawled data is regularly imported into the business application system through python scripts. At the initial stage of the project, the distributed architecture is used as officially recommended by pyspider. Later, it is optimized as the clustered architecture in order to adapt to the actual network environment and the characteristics of crawler tasks. After testing, the work efficiency and stability of the improved architecture have been greatly improved and the expected results have been achieved.