Design and Implementation of a Distributed Crawler System Based on Degree-rank for P2P IPTV
Wenxian Wang · Journal of Sichuan University · 2014
In order to develop a high-performance crawler application for the active measurement of P2P IPTV,based on degreerank to capture peers' information,a new algorithm was proposed. Firstly,a similar strategy of breadth-first search( BFS) was adopted to undertake the initial crawling of peers' information in the overlay. Subsequently,the captured peer list was sorted according to the size of peer's degree. Finally,the sorted peer list was provided to distributed crawler system as the input so as to start the real-time collection of peers' information. The experiment results showed that the performance of the proposed distributed crawler system is better than other existing crawler systems. Specifically,the distributed crawler can track a very popular channel with about 7 200 online peers in about 30 seconds. The results can provide a good solution for capturing more accurate snapshots of peers' information and conducting active measurement study.