Information mining system design and implementation based on web crawler
Shan Lin, You-meng Li, Qingcheng Li · 2008
With the information explosion causing by the World Wide Web in recent years, the issue of how to execute the enormous information efficiently at a reasonable lost has become the concern of information providers, service agencies and end users. When many research focus on how to design an efficient Web crawler, we pay our attention to how to make the best of the result of Web crawler. In this paper, we describe the design and implementation of an information mining system running on the results of Web crawler to gain more metadata from unstructured documents for focused search (such as RSS search). We present the software architecture of the system, describe efficient techniques for achieving high performance and report preliminary experimental results to prove that this system can address the issue of robustness, flexibility and accuracy at a low cost.