Multi-Agent System based Focused Crawling Using Combination of Content and Link Structure Similarity
Ty Rasmey, Dongwoo Lee · Proceedings of KIIT Summer Conference · 2012
This paper introduces an approach focused on topic-specific web crawler that discovers relevant topic with the use of intelligent multi-mobile agent technology is present. We extract topic keyword and judge the topic similarity of a page using combination of link structure which is calculated based on metadata of hyperlink and page content. It also presented a method that allow focused crawler go through several irrelevant pages to get to relevant one when the current page is irrelevant. The applications of system focus on two issues i.) Page relevance to a particular domain that need to crawl resource of relevant topic with high quality, ii) distribute computation among data sites, hence overcoming the drawback of traditional approaches.