Implementation of Efficient Distributed Crawler through Stepwise Crawling Node Allocation
Hyuntae Kim, Junhyung Byun, Yoseph Na, Yuchul Jung · JOURNAL OF ADVANCED INFORMATION TECHNOLOGY AND CONVERGENCE · 2020
Various websites have been created due to the increased use of the Internet, and the number of documents distributed through these websites has increased proportionally. However, it is not easy to collect newly updated documents rapidly. Web crawling methods have been used to continuously collect and manage new documents, whereas existing crawling systems applying a single node demonstrate limited performances. Furthermore, crawlers applying distribution methods exhibit a problem related to effective node management for crawling. This study proposes an efficient distributed crawler through stepwise crawling node allocation, which identifies websites