The crawling and analysis of agricultural products big data based on Jsoup
Jie Wang, Shuo Yang, Yuezhi Wang, Cheng Han · 2015
This paper mainly aims to solve the problem that it is hard to keep a balance between the agricultural production and market, caused by the deficiency of the current circulation of the agricultural products information. We propose and realize a platform that includes the data crawling, information extraction and data analysis of agricultural products big data from Internet. In our system, the data crawling and extraction takes Jsoup, a HTML collector as fundamental technology. According to the website structure, the content is subjected to the analytical process. After correlation analysis and regression analysis, we study on the related factors of price changes, agricultural products market price trends and other issues to get valuable market information available to consumers and help them find cost-effective products. Our work helps to increase the common awareness of agricultural products market, develop agricultural business and boost agricultural e-commerce industries.