A web crawler-based method for collecting information on investment promotion enterprises
Jingyao Sun, Shengnan Zhang, mengli Dai · 2023
Data acquisition is a prerequisite for performing big data analytics. However, as the diversity and timeliness of data increase, the complexity of data collection also increases. In this paper, we take enterprise data on a big data investment platform as the research object, and design two data collection models, static data collection based on incremental crawlers and dynamic data collection based on query topic crawlers, for the static and dynamic characteristics of this data. In the experiments, this paper tests the effectiveness of these two web crawler methods and proves that they can collect static and dynamic investment data comprehensively and accurately. Thus, this study provides an effective data collection scheme that helps improve the accuracy and reliability of big data analysis.