Analysis of Web Spam Structure Using Recursive Strongly Connected Component Decomposition
Youngjoo Chung, Masashi Toyoda, Masaru Kitsuregawa · 2009
Addressing Web spam is a critical issue for today’s search engines. In this paper, we studied a structure of the Web spam using recursive strongly connected component (SCC) decomposition. Spam pages are likely to construct densely connected structures; consequently, SCC decomposition would catch the spam structure of the Web efficiently. Also, by recursive SCC decomposition algorithm with node filtering, denser spam structures would be extracted. We applied SCC decomposition algorithm to our Japanese Web archive crawled in 2004, and separate many large components from a core, the largest component. After this, SCC decomposition algorithm performed again to nodes in the core which have degrees over a threshold. We did this decomposition recursively with increasing threshold. As a result, we found out large link farms in each iteration and this trend continues until at least 10 iterations. We investigated large links farms whether they are spam or not by their URL characteristics. The result showed almost large link farms are spam. Keyword Web spam, Link analysis, Strongly connected components