URL Mining Using Web Crawler in Online Based Content Retrieval
M. Vijayalakshmi, P. Senthil Kumar, M. E. Student, S. Veerasamy · 2014
A supervised web scale forum crawler is a crawling process of forum crawler under supervision(Focus). The main aim of Focus is to crawl related content from the web with minimal overhead and also detect the duplicate links.Forums can contain different layouts or styles and are powered by a variety of forum software packages. Focus take six path from entry page to thread page. It helps the frequent thread updating in forum. It's main purpose is reduce the web forum crawling problem to a URL-type recognition problem.The Focus consists of two parts learning part and online crawling part.The learning part is automatically constructed URL training sets and then online crawling part to crawl all threads efficiently. The accurate and effective regular expression patterns of implicit navigation paths from automatically created training sets using aggregated results from weak page type classifiers.An effective forum entry URL discovery method to ensure the high coverage. The forum crawler should start crawling forum pages from forum entry URLs to thread URLs. The implicit EIT-like path also apply to other User Generated Content (UGC).