Mapping the Blogosphere with RSS-Feeds
Justus Broß, Matthias Quasthoff, Philipp Berger, Patrick Hennig, Christoph Meinel · 2010
The massive adoption of social media has provided new ways for individuals to express their opinions online. The blogosphere, an inherent part of this trend, contains a vast array of information about a variety of topics. It is thus a huge think tank that creates an enormous and ever-changing archive of open source intelligence. Modeling and mining this vast pool of data to extract, exploit and describe meaningful knowledge in order to leverage (content-related) structures and dynamics of emerging networks within the blogosphere is the higher-level aim of the research presented here. This paper focuses on this project's initial phase, in which the above-mentioned data of interest needs to be collected and made available offline for further analyses. Our proprietary development of a tailor-made feed-crawler meets exactly this need. The main concept, the techniques and the implementation details of the crawler thus form the main interest of this paper and furthermore provide the basis for future project phases.